In the never-ending quest to produce high-quality software, traditional dynamic testing plays a fundamental role. The weakness of dynamic testing is that it is only as good as the test cases. To be effective, a great deal of effort must go into writing or generating good test cases, and doing so can be very expensive.
Recently, a new breed of static analysis tools has emerged that can find flaws without writing any test cases. These tools, which are also referred to as static testing tools, can find bugs that are difficult or impossible to find using standard testing methodologies. They can locate serious flaws such as buffer overruns, null pointer dereferences, resource leaks, and race conditions. Because they operate by analyzing the source code itself in detail, they can also highlight inconsistencies or contradictions in the code such as unreachable code, useless assignments, and redundant conditions. Such issues often indicate programmer confusion, and correlate well with bugs. Moreover, knowledge of these issues can actually make writing test cases easier.
There is no substitute for rigorous testing at many levels: unit, regression, functionality and integration testing are all essential, but testing effectiveness depends on the quality of the test cases used. The best test suites are those that have good code coverage. Statement coverage and condition coverage are the most commonly used metrics. Full condition coverage is considered essential for safety-critical code, such as flightcontrol software. Achieving full coverage can be exceedingly time-consuming and expensive. Also, as all programmers know, just because a statement is executed in a successful test case does not mean it will always execute correctly. It may fail under an unusual combination of circumstances that the test cases did not explore.
This is where the strength of static analysis shines through. Static analysis examines paths and considers conditions and variables in the abstract. As such, it can achieve much higher coverage than is usually feasible with dynamic testing. The sooner a bug can be found, the cheaper it is to fix. Static analysis can be used as soon as the code compiles, so it can find flaws before the program is even complete. This is usually much less expensive than writing a test case or debugging a crash.
Figure 1 shows an example of a simple flaw that was found in an opensource program.
In this example, the flaw will only show up in cases of low memory availability, and only when the first allocation succeeds and the subsequent one fails. This is precisely the kind of unusual combination of circumstances that it is hard to reproduce with traditional testing, but static analysis is good at finding.