Ending Spam: Bayesian Content Filtering and the Art of Statistical Language Classification
Jonathan A. Zdziarski
No Starch Press, 2005; 287 pp.; $39.95; ISBN 1593270526
Anyone who has managed to avoid being flooded with spam should consider himself lucky. For everyone else, there is Jonathan Zdziarski's Ending Spam: Bayesian Content Filtering and the Art of Statistical Language Classification. Though the title may be intimidating, the content is both comprehensible and useful.
Zdziarski gives an account of the techniques employed by spammers and filter writers in their arms race beginning with the first spam in 1978. Listing the challenges encountered by both sides of the war makes it abundantly clear that filter writers currently have the upperhand and are widening their lead. To show this, he discusses techniques for obtaining valid test results and provides measures of accuracy for existing filters. Zdziarski uses these results to dispel various myths that keep users from better spam-filtering technology.
The real beauty of Ending Spam, though, is that it provides detailed explanations of filtering techniques without overwhelming or boring casual readers. For these people, short introductions to fundamental ideas such as Hidden Markov Models and Bayes' theorem are provided. Frequent examples help convey core concepts making up for the few instances where explanations are unclear. System administrators and filter authors should also find this book useful, as its explanation of what various features and settings of spam-filtering products can be helpful. Such readers may be particularly interested in string tokenization for language classification; filter training modes; training speed; obtaining good training corpora; and ensuring reasonable space and time efficiency on existing systems.
While many of the concepts covered in this book are not specific to spam filtering, additional challenges posed by spammers cause use of conventional implementations to fail. Zdziarski not only presents current approaches to overcoming spammer tactics used to evade each of the solutions mentioned, but also explains why such attempts ultimately fail. Lastly, he examines various spam-filtering packages currently available including his own, DSPAM, and lists the approaches employed by each. The all-encompassing nature of the book should therefore earn it a space on the shelf of anyone that wants to actively contribute to the reduction of spam.
—J.L.
