Channels ▼

Jonathan Erickson

Dr. Dobb's Bloggers

Pattern Matching as a Literary Pursuit

December 15, 2009

 Pattern matching is one thing computers do really well. Take a bushel of data, throw it at a computer, and be amazed by the results. That's more or less what Sebastian Bernhardsson, Luis Enrique Correa da Rocha, and Petter Minnhagen, all members of Sweden's Umea University Department of Physics, did anyway. What is particularly interesting about their data was that it was a collection of books by Thomas Hardy, D.H. Lawrence, and Herman Melville -- authors I haven't thought much about since university.

And what the researchers discovered was that by applying statistical analysis and pattern matching techniques to literature written by these authors, they were able to identify unique literary "fingerprints" of each author.

More than 75 years ago, George Kingsley Zipf, a Harvard linguist who studied statistical occurrences in different languages, formulated what's known as "Zipf's Law," which states that while only a few words are used very often, many or most are used rarely. More precisely, Zipf suggested that the frequency ranking of a word is inversely proportional to its occurrence. However, this new research suggests that word frequency is less universal than Zipf thought and has more to do with an author's linguistic ability than any over-arching linguistic rule.

The researchers first found that the occurrence of new words in the texts by Hardy, Lawrence, and Melville did begin to drop off in their texts as the book gets longer, despite new settings and plot-twists. They also found that the rate of unique word drop-off varies for different authors and, most significantly, is consistent across the entire works of any one of the three authors they analyzed.

Using statistical patterns from their study, the researchers came up with the idea of a "meta-book," which uses the frequency with which authors use new words in their literature to find distinct patterns in authors' written styles. As they describe in The Meta-book and Size-dependent Properties of Written Language, "these findings lead us towards the meta book concept -- the writing of a text can be described by a process where the author pulls a piece of text out of a large mother book (the meta book) and puts it down on paper. This meta book is an imaginary infinite book which gives a representation of the word frequency characteristics of everything that a certain author could ever think of writing."
 

Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 


Video