Channels ▼
RSS

.NET

Secure Software Needs Careful Testing -- And Lots Of It


Herbert H. Thompson is chief security strategist for People Security, a security education company. He also teaches software security at Columbia University in New York.


Software testing comes in many flavors. Unit testing analyzes individual components before they're integrated into larger systems. System and integration testing checks that modules work together. Regression testing verifies that everything still works after a change is made to the code. And security testing checks that data is protected.

Tools such as source-code scanners, security-aware compilers, and application scanners help developers find vulnerabilities in code. And techniques like fuzz testing uncover inputs that can cause apps to behave badly.

With fuzzing, we deliberately attack software with random data in search of weaknesses and unexpected responses. Fuzz testing is particularly important in Web application development, and it's playing a growing role in ensuring that "security quality" -- the confidentiality, integrity, and availability of systems and data for users -- is integrated into every phase of development.

Fuzz testing finds vulnerabilities that are too complicated for a person to see. With complex programs -- Windows 7 has an estimated 50 million lines of code, for instance -- it's difficult to anticipate everything that can go wrong. Fuzzing takes a "big hammer" approach to systems, hitting them persistently with strange inputs. Fuzzers also save companies money because they don't require much human intervention. Microsoft fuzzes its apps until end of life because it's so cheap to do. Another advantage of fuzz testing is its simplicity. Tests that target specific vulnerabilities can quickly be implemented. Fuzzing is credited recently with finding vulnerabilities in the Windows 7, iPhone, and Android communications protocols.

Early Fuzzing

My first fuzzing experience was in the late 1980s when soda vending machines appeared in the Bahamas, where I grew up. But the machines had signs on them saying "U.S. Quarters Only." The Bahamian currency is pegged one-to-one with the U.S. dollar, and both currencies are accepted everywhere. When my friends and I came across our first soda machine, none of us had a U.S. quarter, so we started experimenting. One of us put a washer in the slot. It dropped right through. We tried a U.S. dime. Nothing. A Bahamian quarter. Still nothing. The outlook for soda was bleak until we dropped a Bahamian 10-cent piece in.

To our amazement, the display instantly registered "25" -- the machine mistook the 10-cent coin for a U.S. quarter. We scrambled to find three more 10-cent pieces. The machine read "50," "75," and finally "1.00." We hit Select and got a soda.

Years later, I learned that the vending machine measured diameter and weight to determine if an object dropped into the slot was a U.S. quarter. Bahamian 10-cent pieces coincidently have almost exactly the same diameter and weight as U.S. quarters.

Without understanding the mechanics of the machine, we had applied semi-random inputs and hoped for something interesting to happen. What we were doing is now known as "fuzzing" -- applying inputs with some degree of randomness, then looking for unexpected outputs. In theory, it sounds straightforward, but in practice, it can be difficult to do well.

How Fuzzing Works

There are many freely available fuzzers, such as Peach Fuzzer from Eddington and Frantz, Spike from Immunity, and MiniFuzz from Microsoft, as well as commercial fuzzers such as Codenomicon from Codenomicon Defensics and Beyond Security's BeStorm. Web application fuzzers are by far the most mature. They understand how an application is supposed to respond to normal input, then they either look for deviations or symptoms of failure. When testing for SQL Injection vulnerabilities, for instance, an input might be a string of text with a single quote mark. If a database error message is returned, the tester knows there's a potential vulnerability. For non-Web software, fuzzing is murkier. These fuzzers work by corrupting files, network traffic, or API parameters by doing the following:

  1. They generate a random input (or randomly corrupt an existing input).
  2. Deliver that input to the app.
  3. Monitor it for an exception or crash.

This approach finds problems such as buffer overflows, a symptom of which is usually a crash. If you're looking for other, noncrashing vulnerabilities, fuzzing can still be useful if it can be automated.

Back to my soda machine, we were looking for one thing -- the red display showing the money we put in had registered. In retrospect, there were other outcomes that would have been interesting: The machine might have crashed, short-circuited, or become hopelessly jammed. If you can specify the symptoms you're interested in, fuzzing can be an incredibly cheap way to see how a system responds to a large set of inputs. The problem is that it can miss more subtle architectural problems.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video