Pattern Matching as a Literary Pursuit
Pattern matching is one thing computers do really well. Take a bushel of data, throw it at a computer, and be amazed by the results. That's more or less what Sebastian Bernhardsson, Luis Enrique Correa da Rocha, and Petter Minnhagen, all members of Sweden's Umea University Department of Physics, did anyway. What is particularly interesting about their data was that it was a collection of books by Thomas Hardy, D.H. Lawrence, and Herman Melville -- authors I haven't thought much about since university.
And what the researchers discovered was that by applying statistical analysis and pattern matching techniques to literature written by these authors, they were able to identify unique literary "fingerprints" of each author.
More than 75 years ago, George Kingsley Zipf, a Harvard linguist who studied statistical occurrences in different languages, formulated what's known as "Zipf's Law," which states that while only a few words are used very often, many or most are used rarely. More precisely, Zipf suggested that the frequency ranking of a word is inversely proportional to its occurrence. However, this new research suggests that word frequency is less universal than Zipf thought and has more to do with an author's linguistic ability than any over-arching linguistic rule.
The researchers first found that the occurrence of new words in the texts by Hardy, Lawrence, and Melville did begin to drop off in their texts as the book gets longer, despite new settings and plot-twists. They also found that the rate of unique word drop-off varies for different authors and, most significantly, is consistent across the entire works of any one of the three authors they analyzed.
Using statistical patterns from their study, the researchers came up with the idea of a "meta-book," which uses the frequency with which authors use new words in their literature to find distinct patterns in authors' written styles. As they describe in The Meta-book and Size-dependent Properties of Written Language, "these findings lead us towards the meta book concept -- the writing of a text can be described by a process where the author pulls a piece of text out of a large mother book (the meta book) and puts it down on paper. This meta book is an imaginary infinite book which gives a representation of the word frequency characteristics of everything that a certain author could ever think of writing."