Herbert H. Thompson is chief security strategist for People Security, a security education company. He also teaches software security at Columbia University in New York.
Software testing comes in many flavors. Unit testing analyzes individual components before they're integrated into larger systems. System and integration testing checks that modules work together. Regression testing verifies that everything still works after a change is made to the code. And security testing checks that data is protected.
Tools such as source-code scanners, security-aware compilers, and application scanners help developers find vulnerabilities in code. And techniques like fuzz testing uncover inputs that can cause apps to behave badly.
With fuzzing, we deliberately attack software with random data in search of weaknesses and unexpected responses. Fuzz testing is particularly important in Web application development, and it's playing a growing role in ensuring that "security quality" -- the confidentiality, integrity, and availability of systems and data for users -- is integrated into every phase of development.
Fuzz testing finds vulnerabilities that are too complicated for a person to see. With complex programs -- Windows 7 has an estimated 50 million lines of code, for instance -- it's difficult to anticipate everything that can go wrong. Fuzzing takes a "big hammer" approach to systems, hitting them persistently with strange inputs. Fuzzers also save companies money because they don't require much human intervention. Microsoft fuzzes its apps until end of life because it's so cheap to do. Another advantage of fuzz testing is its simplicity. Tests that target specific vulnerabilities can quickly be implemented. Fuzzing is credited recently with finding vulnerabilities in the Windows 7, iPhone, and Android communications protocols.
Early Fuzzing
My first fuzzing experience was in the late 1980s when soda vending machines appeared in the Bahamas, where I grew up. But the machines had signs on them saying "U.S. Quarters Only." The Bahamian currency is pegged one-to-one with the U.S. dollar, and both currencies are accepted everywhere. When my friends and I came across our first soda machine, none of us had a U.S. quarter, so we started experimenting. One of us put a washer in the slot. It dropped right through. We tried a U.S. dime. Nothing. A Bahamian quarter. Still nothing. The outlook for soda was bleak until we dropped a Bahamian 10-cent piece in.
To our amazement, the display instantly registered "25" -- the machine mistook the 10-cent coin for a U.S. quarter. We scrambled to find three more 10-cent pieces. The machine read "50," "75," and finally "1.00." We hit Select and got a soda.
Years later, I learned that the vending machine measured diameter and weight to determine if an object dropped into the slot was a U.S. quarter. Bahamian 10-cent pieces coincidently have almost exactly the same diameter and weight as U.S. quarters.
Without understanding the mechanics of the machine, we had applied semi-random inputs and hoped for something interesting to happen. What we were doing is now known as "fuzzing" -- applying inputs with some degree of randomness, then looking for unexpected outputs. In theory, it sounds straightforward, but in practice, it can be difficult to do well.
How Fuzzing Works
There are many freely available fuzzers, such as Peach Fuzzer from Eddington and Frantz, Spike from Immunity, and MiniFuzz from Microsoft, as well as commercial fuzzers such as Codenomicon from Codenomicon Defensics and Beyond Security's BeStorm. Web application fuzzers are by far the most mature. They understand how an application is supposed to respond to normal input, then they either look for deviations or symptoms of failure. When testing for SQL Injection vulnerabilities, for instance, an input might be a string of text with a single quote mark. If a database error message is returned, the tester knows there's a potential vulnerability. For non-Web software, fuzzing is murkier. These fuzzers work by corrupting files, network traffic, or API parameters by doing the following:
- They generate a random input (or randomly corrupt an existing input).
- Deliver that input to the app.
- Monitor it for an exception or crash.
This approach finds problems such as buffer overflows, a symptom of which is usually a crash. If you're looking for other, noncrashing vulnerabilities, fuzzing can still be useful if it can be automated.
Back to my soda machine, we were looking for one thing -- the red display showing the money we put in had registered. In retrospect, there were other outcomes that would have been interesting: The machine might have crashed, short-circuited, or become hopelessly jammed. If you can specify the symptoms you're interested in, fuzzing can be an incredibly cheap way to see how a system responds to a large set of inputs. The problem is that it can miss more subtle architectural problems.
