Tracking, Code Coverage, and Metrics
Code coverage is a term referring to the amount of process state a fuzzer induces a target's process to reach and execute. At the time of writing, we are currently unaware of any publicly or commercially available fuzzing technology capable of tracking and logging code coverage. This is an important concept for analysts across the board. Quality assurance (QA) teams can utilize code coverage as a metric to determine confidence in the level of testing that has taken place. If you are the QA lead for a Web server product, for example, you would probably feel more comfortable shipping your product with zero failures across 90 percent code coverage than you would with zero failures across only 25 percent code coverage. Vulnerability researchers can benefit from code coverage analysis by identifying the modifications necessary to expand their code coverage into more obscure states of their target where other eyes may not have already been.
Think about creative approaches to determining code coverage and the benefits that such an analysis might provide as we discuss various fuzzers in upcoming chapters. When fuzzing, people always ask,"How do I start?" Remember that it's equally important to ask, "When do I stop?"
Error Detection
Generating and transmitting potentially malicious traffic encompasses only half of the fuzzing battle. The other half of the battle is accurately determining when an error has occurred. At the time of writing, the majority of available fuzzers are "blind" in that they have no concept of how the target reacts to transmitted tests. Some commercial solutions interweave "ping" or keepalive checks between malicious attempts as a control to determine whether or not the target is still functional. The term ping here is loosely used to refer to any form of transaction that should generate a known good response. Other solutions exist that build on log output analysis. This could involve monitoring ASCII text logs maintained by individual applications or querying entries in system logs such as the Windows Event Viewer as shown in Figure 6.
The benefit of these approaches to error detection is that they are, for the most part, easily ported between platforms and architectures. However, these approaches are severely limited with regard to the kinds of errors they are capable of detecting. Neither of these approaches, for example, can detect the case where a critical error occurs in a Microsoft Windows application but is gracefully handled by a Structured Exception Handling4(SEH) routine.
The next generation in error detection is the use of lightweight debug clients to detect when an exceptional condition has occurred in a target. The one negative aspect of utilizing these types of tools is that you have to develop one for each target platform on which you are testing. For example, if you want to test three SMTP servers on Mac OS X, Microsoft Windows, and Gentoo Linux, you will likely have to develop two or possibly three different monitoring clients. Furthermore, depending on your target, it might not be possible or timely to construct a debugging client. If you are testing a hardware VoIP phone, for example, you might have to fall back to control testing or log monitoring, as hardware solutions are less conducive to debugging and might require special tools.
Looking even further ahead, the panacea of error detection lies in dynamic binary instrumentation/translation5 (DBI) platforms such as Valgrind and Dynamo Rio. On such platforms, it becomes possible to detect errors as they develop rather than after they trigger. At a 50,000-foot view, DBI-based debugging engines are able to very specifically analyze and instrument a target software application at a low level. Such control allows for the creation of memory leak checks, buffer overflow and underrun checks, and so on. Referring to the memory corruption example we used when discussing reproducibility, a lightweight debug client is capable of informing us when the memory corruption triggers. If you recall our example, we posed a scenario whereby a number of packets were sent to the target service, which crashed on receiving the 50th test case. On a platform such as Valgrind, we might be able to detect the initial memory corruption that occurred at some earlier test prior to triggering the exception. This approach can save hours and perhaps days of fuzz tuning and bug tracking.
Various nontechnical factors such as budget and deadlines can impose limitations on fuzz testing. These factors must be kept in mind during the design and planning stage. You could, for example, find yourself in a last-minute, prelaunch panic because no one has even briefly examined the security of your $50-million product investment. Security is all too often an afterthought in the software development lifecycle (SDLC), taking a backseat to adding new features and meeting production deadlines. Security must be "baked in" as opposed to being "brushed on" if we ever hope to produce secure software. That involves making fundamental changes to the SDLC to ensure that security is considered at every stage of development. That said, we recognize that software is developed in the real world and not utopia where resources are plentiful and defects are scarce. As you progress through this book it is therefore equally important to mentally classify various techniques that can be applied when time and finances are limited as well as dreaming up the "ultimate" fuzzing suite. Consider also where you would implement such tools in the SDLC and who would be responsible for owning the processes.
