Channels ▼
RSS

Testing

Deploying Static Analysis


Static source code analysis for bug finding ("static analysis" for short) is the process of detecting bugs via an automated tool that analyzes source code without executing it. The idea goes back at least to Lint, which was invented at Bell Labs in the 1970s, but static analysis has undergone a revolution in effectiveness and usability in the last decade. The initial focus of static analysis tools was on the C and C++ programming languages. Such tools are particularly necessary given C/C++'s notorious flexibility and susceptibility to low-level bugs. More recently, tools have flourished for Java and/or Web applications; these are needed because of the prevalence of easily exploitable network vulnerabilities. When using such tools, it is all too easy to deploy them in a way that looks good superficially, but misses important defects, shows many false positives, and brings the tool into disrepute. This article is a guide to the process of deploying a static analysis tool in a large organization while avoiding the worst organizational and technical pitfalls.

Leading commercial static analysis tools with which I am familiar include Coverity, Fortify (now owned by Hewlett-Packard), and Klocwork. Klocwork and Coverity both initially focused on C/C++, although they came from opposite origins: Klocwork from the telephone equipment company Nortel, and Coverity from Stanford University. Fortify's initial focus was on security for Web applications in languages such as Java and PHP. All three companies are now encroaching on each others' territories, but it remains to be seen how well they will do outside of their core competencies.

An excellent, free, but limited, academic static Java byte code analysis tool is FindBugs. Its lack of an integrated database for defect suppression makes its large-scale use difficult in sizable organizations, but its use by individual developers within the Eclipse development environment can be extremely valuable. Similarly, recent versions of Apple's Xcode and Microsoft's Visual Studio development environments contain integrated static analysis tools for C/C++. These are useful for finding relatively shallow bugs while an individual developer is writing code; their short feedback loop bypasses the difficulties of broader deployment of tools that perform deeper analysis. A longer list of tools is provided in Figure 1 (which is taken from "Magic Quadrant for Static Application Security Testing," by Gartner Inc.), albeit with a strong bias towards security and adherence to Gartner's strategy recommendations.

Figure 1.

There is no general-purpose introductory textbook on the subject; the best general introduction is a short article by Dawson Engler, the inventor of Coverity, et al. Two of the leaders at Fortify have written an introductory textbook, but it focuses primarily on their tool and on security, and skimps on key static analysis concepts. There is a rigorous academic textbook on the more general concept of static analysis, but it preceded the revolution in static analysis for bug finding.

Getting Started: The Politics

The first question to ask before deciding to do static analysis in an organization is not what tool to buy, nor even whether you should do static analysis at all. It's "why?"

If your purpose is genuinely to help find bugs and get them fixed, then your organizational and political approach must be different from the more usual, albeit unadmitted, case: producing metrics and procedures that will make management look good. (Fixing bugs should actually be your second goal: An even higher goal is preventing bugs in the first place by making your developers learn from their mistakes. This also contraindicates outsourcing the evaluation and fixing of defects, tempting though that may be.)

Political Issues to Settle in Advance

Get buy-in from your testing/quality assurance department. They must support the project and will have authority over quality-related issues, even if they inconvenience the other stakeholders. Quality has a much smaller constituency than the schedule or the smooth running of internal procedures, but it must be the final arbiter for crucial quality-related decisions (see chapter 22 of Joel On Software for more on this topic).

Give some thought to what part of the organization, if any, should be in charge of running the tool once it's set up. If your organization has a tools team, it might seem the obvious owner, but this does need careful consideration. Static analysis for bug finding is probably not your organization's core competency, and you will need to worry about the Iron Law of Bureaucracy: Your tools team's institutional interest will be in the smooth running of the tool, not in the messy changes necessary for finding bugs. Even if you're reluctant to outsource the rest of the process, administration and configuration may be more flexible if done by external players rather than an internal team with its own interests, habits, and procedures. It may also be more productive to hire an expensive consultant for a few hours, rather than a lesser-paid internal resource full-time; an external resource may be more flexible and less prone to establishing an entrenched bureaucracy.

Engineering Management

The conventional wisdom is that getting most developers to use a static analysis tool requires a high-ranking management champion to preach its benefits, ensure that the tool is used, and keep the focus on finding bugs and getting them fixed. The flip side to this is that any attempt to herd cats (or to get programmers to adhere to best practices) will cause a backlash. Your tool must withstand scrutiny from developers looking for excuses to stop using it.

Get buy-in from engineering that they will make time in the schedule to review and fix bugs found, even if they are disinclined to do so, which they will be once they see the first false positive. (Or even the first false false positive; more on this later.) Ensure that it's not the least-effective engineers whose time is allotted for reviewing and fixing static analysis bugs (more on this momentarily). You'll also need agreement from the security team that sales personnel will get access to your real source code; more on that is also to follow.

Smart Programmers Add More Value — and Subtract Less

Handling static analysis defects is not something to economize on. Writing code is hard, finding bugs in professional code should be hard, and evaluating possible mistakes in alleged bugs is even harder. Learning to evaluate static analysis defects, even in a developer's own code, requires training and supervision. It is necessary to tread delicately around the polite pretense that the code owner is an infallible authority on the behavior of that code. Misunderstandings about the actual behavior of unsigned integers and assertions are, for instance, regrettably common in my experience.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Comments:

ubm_techweb_disqus_sso_-44a77e3484b420435d7892582488dc5f
2012-08-12T17:35:07

I also propose to try PVS-Studio (Static Code Analyzer for C/C++/C++11) - http://www.viva64.com/en/pvs-s...


Permalink
ubm_techweb_disqus_sso_-6d80205e22b581b323cb4b70d537756d
2012-08-01T03:34:23

This is as great artilcle, summarizing the challenges of deployment and the values.

Frank


Permalink
ubm_techweb_disqus_sso_-40faeb4ffcfd5c655b04301c06fc4271
2012-07-16T21:14:02

Loved the article! Thanks!


Permalink

Video