This was the last post I made to the old Typepad-hosted Dabbler.

Here mainly for historical purposes.

One of the unanswered questions is what the thieves do with all the files this tool sends back to the FTP server.  If they were to capture everything on my work PC, they'd get a middle manager's desktop conglomeration:  Thousands of more-or-less useless files and a few dozen documents which would probably be valuable to someone.  There are also a bunch of database files (in several formats) which could be mined, but aren't clearly useful as they stand.  I have trouble finding stuff in this mess; how would someone who doesn't work here make sense of it?

Not that it couldn't be done; it's what spies do for a living, after all.  But I'm interested in how these gals(?) approach the problem. Might be something of value for us civilians.

Link courtesy of F-Secure's blog.

Thanks, as is often the case, to SANS ISC for the link.

Here's a summary of the last three months usage, in the format I've used on prior reports:

The test period ended July 3, 2004, at 7,292 messages.

• 168 (2.3 %) were sent to the wrong bucket.
  • (Therefore) 97.7 % were sent to the right bucket.
  • This percentage took a significant hit at the start of the baseball season, when a bunch of email sources came back to life.
• 3,397 (46.6%) were spam.  (This is a significant increase, I'd say, from the previous 41.0%.)
  • A handful of these are from legitimate e-mail lists whose owners make it difficult to unsubscribe, but the impact is minimal.  
• Only 11 messages were auction-related; 3 of these were false negatives and 1 was a false positive.  I seem to have stopped hanging around eBay, at least for now.  
• The Vendor (100 messages/16 false +/11 false -) and Mailing List (402/48 f+/8 f-) categories, both of which are catch-alls, seem to show real improvement, though this is still a significant source of error.  The problem continues to be that "well-designed" spam looks superficially like these categories.
• The problem I reported with e-mails from Change Detection still exists and remains annoying, but has improved; basically, PF sees several classes of messages as too similar to differentiate.  It's pretty clear to me that the algorithm isn't looking at the problem the way I think it should.

Every now and then a spammer finds a hole in this defense, but after a couple days PF has things sorted out again.  That's how things should work.

For the record, I'm currently using POPFile version 0.20.1, which uses the BerkeleyDB for storage.  The developers moved to a SQL engine in March with version 0.21.0 (currently 0.21.1), but didn't convince me a change was necessary; I'm unlikely to change until there's a major upgrade.   Version 0.20 is slower than version 0.19 was, but not in ways which bother me.  Your mileage may vary, of course.

Thus my current report.  I remain very satisfied with the tool.

Team Cymru

Witty's network impact was really quite dramatic.  Cymru, in an article aimed mainly at network administrators, shows the worm's very dramatic traffic increase  as recorded on three servers--look for graphs about a quarter of the way into the essay.  Worth a look, even if (like me) you can't make more than superficial sense of most of the discussion.  That propogation pattern is remarkable; anyone who can make a program launch look like that deserves our respect..

There's a mind behind Witty, folks.  Wonder what she learned from the experiment.

Not that there are no other fish in the ocean upon whom I can sling my hook, but who can be like my dear Betsy that loves me with such generosity of heart?

---found in Ian's bottle....

Ian Shoales has been reading his spam, and reports. (Originally found in Intelligent Enterprise magazine, which is no more....)

Colleen Shannon and David Moore of the Cooperative Association for Internet Data Analysis (CAIDA) offer a fascinating and rather frightening analysis of last week's Witty worm.  It's dangerous out there, folks.

Witty seems to have been an orchestrated attack, albeit using an opportunistic method.  The paper argues convincingly that we're using the wrong security model; if we don't change, the bad guys are gonna take down a lot of computers.  Doesn't matter if it's a prank or something really malicious, it's going to be costly regardless.

Link courtesy of Mikko Hyppönen/F-Secure

There used to be a technical analysis of Witty, from Matthew Murphy, but it seems to be gone. It was also worth a read, if you didn't mind slogging through code.

Continuing in the same format I used in my earlier note about PF:

Fourth Thousand

This test span ended November 21 at 1,000 messages.

• 26 (2.6%) were sent to the wrong bucket.
  • 97.4% were sent to the right bucket....
• 415 (41.5%) were spam. Again: Wow!
• 4 (0.4%) were probably virus-laden.
• 4 (0.4%) were bounced email.
• Auction seems to be fully solved; 39 messages, with one false positive and one false negative.
• The Vendor category may have finally improved: 14 messages; only three errors.
• Lists looks better: Ten false positives and no false negatives associated with 51 messages.
• A new category, created (with my new e-mail address) to service this weblog, had 8 errors--to go with seven messages. New categories are always problems....

All in all, that's a rather impressive performance. The increasing spam count is also rather impressive; after all, I added this tool to my kit because the junk seemed to be getting out of hand.

The new POPfile version has a thoroughly-revamped back end, and some modifications to the code in the engine. We'll see how it goes.

Jon Udell's also talking about using Bayesian categorizers, at both a higher level of abstraction and greater detail. Worth a look.

An essential fact:  While POPfile usually functions as a spam filter, its design supports sophisticated sorting of email into a large number of categories.  I'm using it as a mail sorter; the spam filter is important, but the software's smart about all of my mail, and in a real sense the spam folder's just another target for the sorter.

Basic Information

I receive between 50 and 100 e-mails each day, and read about 60% of those (the unread ones are either duplicates or spam). I used to read about 85% of my mail; the change in percentage is largely because of the increasing spam load. (Eudora has a reporting function; these numbers have some relation to reality.) Perhaps 65% of the real mail has baseball content of some sort or other; the rest is on a wide range of topics.

These get sorted into a couple dozen categories; I tinker with these a bit, but they are essentially the same categories I used for sorting e-mail in 1995.  A large percentage of my mail originates from the Society for American Baseball Research list called SABR-L, which has its own folder; the remaining folders group mail in ways which largely reflect my mental prioritizations.  One folder, called "Lists," is the target for mailing lists on miscellaneous topics.  I sometimes ignore SABR-L for months; I check my eBay mail daily.

After reading the POPfile documentation, I decided to see how well it sorted the total daily package.  I set up "buckets" to match the folders, replaced several hundred Eudora rules with twenty-five, and set about teaching POPfile how to sort things. This story begins on August 18.

Here's my report....

First Thousand

Since you train POPfile by correcting its errors, the first few dozen messages are basically all errors and the first few hundred are unreliable.  I took an accounting after message 1,049, which arrived on September 30.

• 104 (10.0%) were sent to the wrong bucket.
  • 90.0% were sent to the right bucket....
• 207 (19.7%) were spam.
• 25 (2.4%) were probably virus-laden.
• 114 (10.9%) were bounced email.
• PF had particular problems with the Auction bucket; it made 15 wrong guesses (11 false positives & 4 false negatives) in a category with only 11 total messages.
• PF also had significant problems with the Vendor bucket, with eight sorting errors among only nine total messages.
• The List category, which seems to me the most difficult to train, received 40 messages; PF generated 12 false positives and 4 false negatives.

Second Thousand

POPfile weathered its adolescence in the first half of October, and reached message 999 on October 18.

• 41 (4.1%) were sent to the wrong bucket.
  • 95.9% were sent to the right bucket....
• 249 (25.7%) were spam.
• 5 (0.6%) were probably virus-laden.
• 0 (0.0%) were bounced email.
• PF stopped having problems with Auction; 30 messages, with no false positives and three false negatives.
• PF's Vendor bucket issues seemed to abate, with only five sorting errors among twenty-one total messages. Better, but still unacceptable.
• The List category continued about as before: 55 messages, with 12 false positives and 2 false negatives.

Third Thousand

This test span ended November 4 at 1,008 messages.

• 41 (4.1%) were sent to the wrong bucket.
  • 95.9% were sent to the right bucket....
• 408 (40.7%) were spam. Wow!
• 2 (0.2%) were probably virus-laden.
• 2 (0.2%) were bounced email.
• Auction was basically clean; 20 messages, with one false positive and one false negative.
• PF's Vendor bucket sort deteriorated, with thirteen sorting errors among twenty-nine total messages. Yucky.
• The List category remains problematic: 51 messages, with 15 false positives and 3 false negatives. I suspect this will only improve if I split the category into logical sub-groups.

Since November 4

I've received 712 messages; 97.6% are being sorted correctly. Not bad, if you ask me. I'll not give you a further breakdown 'til I reach 1,000.

POPfile's principal author, John Graham-Cumming, announced a new version a couple weeks ago, which I've not yet installed. I'll do that in a day or two.