Channels ▼

Hard Copy: Ending Spam and Managing Managers

Ending Spam: Bayesian Content Filtering and the Art of Statistical Language Classification

Jonathan A. Zdziarski

No Starch Press, 2005; 287 pp.; $39.95; ISBN 1593270526

Anyone who has managed to avoid being flooded with spam should consider himself lucky. For everyone else, there is Jonathan Zdziarski's Ending Spam: Bayesian Content Filtering and the Art of Statistical Language Classification. Though the title may be intimidating, the content is both comprehensible and useful.

Zdziarski gives an account of the techniques employed by spammers and filter writers in their arms race beginning with the first spam in 1978. Listing the challenges encountered by both sides of the war makes it abundantly clear that filter writers currently have the upperhand and are widening their lead. To show this, he discusses techniques for obtaining valid test results and provides measures of accuracy for existing filters. Zdziarski uses these results to dispel various myths that keep users from better spam-filtering technology.

The real beauty of Ending Spam, though, is that it provides detailed explanations of filtering techniques without overwhelming or boring casual readers. For these people, short introductions to fundamental ideas such as Hidden Markov Models and Bayes' theorem are provided. Frequent examples help convey core concepts making up for the few instances where explanations are unclear. System administrators and filter authors should also find this book useful, as its explanation of what various features and settings of spam-filtering products can be helpful. Such readers may be particularly interested in string tokenization for language classification; filter training modes; training speed; obtaining good training corpora; and ensuring reasonable space and time efficiency on existing systems.

While many of the concepts covered in this book are not specific to spam filtering, additional challenges posed by spammers cause use of conventional implementations to fail. Zdziarski not only presents current approaches to overcoming spammer tactics used to evade each of the solutions mentioned, but also explains why such attempts ultimately fail. Lastly, he examines various spam-filtering packages currently available including his own, DSPAM, and lists the approaches employed by each. The all-encompassing nature of the book should therefore earn it a space on the shelf of anyone that wants to actively contribute to the reduction of spam.


Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.