Channels ▼
RSS

Database

Deploying Static Analysis


One common facility is ranking defects by the checker that finds them: However, an unreliable defect with little impact is less important than a definite bug with bad consequences, regardless of which checkers found them. I have no general solution to offer users of existing tools, beyond the advice to follow on experimenting, measuring, and adapting. Don't dismiss even unambitious checkers too hastily. Engler's Third Law is that no bug is too foolish to check for and this is depressingly well confirmed by experience, at least for C/C++. (Egregious bugs may be harder to find in Java.) In code under development, the simplest checks for the most painfully obvious bugs are often the most effective, such as use of a variable immediately after free(), etc.

Handling Static Analysis Defects

Identifying bugs and their fixes via static analysis is the easy part; the tool goes right to the heart of a defect and highlights the problem in clear, bright colors. Difficulties usually arise after the fix is identified, either because it is misunderstood (which I discuss below), or because there is resistance to making the change. Given that a tool is good enough to find real bugs, what determines whether the bugs get fixed is the sociology of the organization and its politics.

The conventional wisdom is that static analysis tools have depressingly little overlap; but missing bugs is not the end of the world. As long as the pipeline is full of genuine and significant bugs, a good static analysis tool will do a lot of good for the code and the customers. Conversely, some genuine bugs found by the tool will be ignored, either as incorrectly deemed false positives, annoyances swept under the rug, or simply not gotten to in time due to the Embarrassment of Riches problem.

Keep a close eye on the ratio of false positives; anything much above 20% indicates a configuration error, user errors, or (more rarely) a bad checker that should be disabled.

Novice static analysis users need to accept that, when they disagree with the tool, they will usually (but not always) be wrong. This is not to say that any tool is perfect; once a user understands the kinds of mistakes it makes (and doesn't make), human supervision can begin to add value. So, do not make the opposite mistake and fix all defects blindly; unnecessary code changes carry their own risks and I have seen mistakes (detected by later static analysis) in code required to silence earlier static analysis warnings. In particular, a strict policy of requiring immediate fixes for newly checked-in static analysis defects might have unintended consequences. A programmer who is in a hurry to get home after a check-in is unlikely to be in the best state of mind for analyzing and fixing static analysis defects.

A grayer area is defects where the tool is technically correct, but the developer believes that the code path is not worth worrying about. This is a judgment call, and hence hard for anyone but the code owner to second-guess, which also makes it impractical to judge statistically.

Practicalities

Most major tools present found defects in a browser. This makes the user interface of your particular browser of considerable importance and necessitates a delicate balance between habitual usability and specific features needed for examining static analysis defects.

Symbol Highlighting

Firefox's Highlight All feature is invaluable for quickly highlighting all occurrences of the currently selected text (such as a variable or function name).

If you will be examining many defects, it's worth practicing the key sequence until it's instinctive. (On Macintosh Firefox, one of the keyboard equivalents is missing, so a keyboard macro program is necessary.)

Most browser-based tools have similar functionality that's symbol-based, which has advantages, disadvantages, and limitations. Some instances of the symbol can be missed, but on the other hand, a symbol that is a superstring of the selected symbol will be properly ignored by symbol-based search, but improperly highlighted by a browser's string-based search.

Quick highlighting can reduce some types of defects from a cognitive problem to simple pattern recognition. For instance, a buffer overrun defect with a parameter array index can often be recognized merely by highlighting the index on the line where the tool reports the defect. The pattern of color between there and the function head makes it obvious where there are checks, if any, on the parameter.

Incompatibilities and Difficulties

Some tools cause difficulties with some browsers. Coverity 5.x's heavy-duty Web pages, for instance, do not work properly with some versions of Firefox and will not open separate pages for function definitions; this is not a problem with Coverity 4.x. (My workaround for the latter limitation is to take quick notes via text drag and drop, but this requires support from both the browser and your text editor.) Fortify's Web interface uses Adobe Flash, which does not support standard text selection behavior, so I recommend avoiding it in favor of their standalone application (which has text interface difficulties of its own.)

Improper configuration, in particular errors in locating and meta-compiling your source, can silently ruin your analysis while leaving the illusion of doing useful work. Build and parsing errors can make most defects false positives, or miss many of the genuine defects the tool is capable of finding. The upside is that a few simple (though perhaps hard-to-find) fixes to the build configuration can convert a worse-than-useless analysis, consisting of mostly false positives, into something worth getting your engineers to look at. But you must set up the procedure so that those configuration fixes are made promptly: Letting a bogus build count as a success or setting up a slow bureaucracy to approve changes will bring the tool into disrepute, and give engineers an excuse to ignore even legitimate bugs found by the tool.

Integration with Defect Tracking

Most commercial tools come with a defect-tracking system, which is essential for suppressing false positives. This is essentially a bug database, but without many essential features, such as status tracking. (The first thing a quality assurance professional checks in the morning is changed bugs in the database, for instance, to see what developers and managers have been marking as unimportant.) None of the bundled databases known to me matches the features of professional bug-tracking databases; I have sometimes been reduced to archiving bug lists as CSV files, and diffing them as time goes by.

Sometimes it is useful to link an organization's real bug-tracking database with a static analysis defect database. I don't advise importing defects automatically. In my opinion, a real bug database should be reserved for issues that a human has judged to be genuine. One strategy I have found useful is to file one bug report per component into your existing bug database, with a link in each bug report to a live query with suspected defects in the component in the current build, with separate counts for high- and low-priority checkers.

Keep Measuring and Adapting

As I have mentioned previously, metrics — applied blindly — make a very bad master; but applied wisely, they can be a helpful servant. Track the numbers of important (and unimportant) defects found, and the checkers that found them. Even good checkers might not find the sort of defects that your organization is, realistically, going to fix. Track the defects that actually are fixed, preferably by keeping an eye on source code management check-ins. (Pretend not to see check-in comments that minimize the importance of the defect fixed; developers are often reluctant to admit that an automated tool found significant bugs in their code. These imperfect developers are your best customers, not your enemy.) Given finite resources, it is crucial to prioritize defects; sadly, this sometimes means ignoring some components or disabling some checkers.

Also keep an eye on static analysis defects that are not fixed. Sometimes the problem will be with the tool, and some checkers are not worth the resources they consume. More often, the problem will be with the reaction to the tool, and sometimes education is necessary. That is perhaps the most valuable result of static analysis: In the long run, making developers think about their code and learn about what it actually does, is even more important than fixing the current version of their code.


Flash Sheridan was a static source code analysis consultant at Bell Labs when writing this article, and has also done static analysis consulting at Qualcomm and Palm. He is currently testing a combined compiler and static analyzer for a major technology company.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video