Whatever happened to the idea of Bayesian spam filtering?

I mean, any idiot these days can tell a SPAM email from a real email, so how on earth do these stupid things keep getting through?

I am not talking about covert advertising cleverly disguised as real text; that may take time. But the vast majority of SPAM is blatant.

I don't like insulting my friends by requiring them to fill in a stupid blank in order to send me an email, nor do I wish to install some sucky-ass software which generates its own SPAM by sending out ads to anyone who gets email from me.

I wonder why the Bayesian software hasn't been developed more fully. Might it just put an end to SPAM as we know it?

It might, but on the other hand it might not! This post (which begins with an "intelligent" SPAM email) demonstrates the problems inherent in the Bayesian approach:

[M]arking such emails as spam will increase the probability of false positives in the future. If you receive a lot of these mails, certain rare words will be associated very highly with spam by your filter. Then, when you get an innocent-seeming email from a friend that happens to contain the words “schizophrenic pompous playwright”, that will be enough to get it black-holed.

Amusingly enough, this email would likely be shot down by a filter because of the method it uses to hide the words in the HTML file. It’s only a first attempt, though. Spammers will get better at it. So long as the spammer’s dictionary is big enough, and they regularly rotate their words, this could be an effective technique to weaken Bayesian filtering on two fronts: increasing both its false-negatives and false-positives.

One problem (one that Graham admits to) is that Bayesian classification assumes that the elements of the object being classified (in this case, the words of the email) are independent of each other. This is a troublesome assumption to make about language, in that the filter can’t tell that the list of words does not match the patterns we would normally recognise as the language of a legitimate email.

On the other hand, if we had a more sophisticated linguistic filter, it wouldn’t be hard for spammers to come up with a program that generated random, but grammatically correct sentences.

It’s a war of escalation.

What just sticks in my craw is how easy it is for humans to spot SPAM, yet how difficult it is for computers. Fuzzy logic somehow has to supply the answer.

Meanwhile, the spammers have nothing but time and fanatic devotion to their silly games of annoyance.

Along similar lines, I often wonder whether anti-virus companies hire virus writers. (I can't think of a more intriguing conflict of interest than the sort of moonlighting which might be officially unapproved, yet guarantee promotion -- for if there are no new viruses you can't sell software! Money plus conflicts of interest means mutually escalated non-destruction -- if that's not too fuzzy.)