Dr. Dobb's Journal February 1997

Static and Dynamic Testing

Static tests are performed without actually executing the program being tested; dynamic tests require execution. Program compilation is a static test; more advanced forms use proven techniques to verify the correctness of a program. Dynamic testing is performed in a crude form by protected-mode operating systems such as UNIX, OS/2, and Windows NT, which terminate an application if it steps outside its allotted address space or instruction repertoire. More advanced forms use various ways of instrumentation to keep a closer watch on the program's behavior.

The pros of static checking are that: full coverage is attainable in theory (but not yet realized in practice); it detects both faults and other problems, such as portability, style, and the like; it is independent of the quality of test cases; and error reports are immediately linked to the actual fault.

The cons of static checking are that: It requires access to source code; in practice, limits to value tracking restrict coverage; and detection of dynamic problems (for example, interactions, synchronization) is limited or absent.

The pros of dynamic checking are that: It detects problems that occur only at run time (for instance, those caused by specific interaction patterns); value tracking and API checking in principle are unlimited; and access to source code is not (always) required.

The cons of dynamic checking are that: coverage is strongly determined by the quality of actual test cases; detected failures may be difficult to relate to actual faults; instrumentation changes the program image and may introduce problems in and of itself; and run-time performance may be reduced and thereby cause problems (in real-time systems, for instance).

Regardless of the testing approach, a number of problems will not be caught. Errors of omission are notoriously hard to detect, as are errors of logic (branching the wrong way), misinterpretation of data values, and erroneous state transitions. Also, verification of a program's function against the requirements, user interface design, and performance testing are well outside the realm of these tools. Therefore, you'd be well-advised not to rely solely on automated testing tools. No matter how useful they are, a clean bill of health from such a tool should be regarded as a necessary, but by no means sufficient, condition to guarantee a correct program.

-- -R.v.d.W.