Mark Pitchford specializes in software test as a Field Applications Engineer with LDRA.
Software test tools have been traditionally designed with the expectation that the code has been (or is being) designed and developed following an ideal development process which adheres to a best practice approach. Such an approach implies the existence of clearly defined requirements, an adherence to corporate or international coding standards, a well-controlled development process, and a coherent test regime.
Legacy code turns the ideal process on its head. Although such code is a valuable asset, proven in the field over many years, it was likely to have been developed on an experimental, ad hoc basis by a series of "gurus" -- experts who prided themselves at getting things done and in knowing the application itself, but not necessarily expert at complying with modern development thinking and bored with providing complete documentation.
Frequently, this legacy software -- software of unknown pedigree ("SOUP") -- forms the basis of new developments which must meet modern coding standards either due to client demands or simply a policy of continuous improvement within the developer organization. The need to leverage the value of SOUP while meeting new standards and further developing functionality presents its own set of unique challenges.
The Dangers of SOUP
Many SOUP projects will have initially been subjected to functional system testing which usually offers very poor code coverage, leaving many code paths unexercised. When that code is applied in service, the different data and circumstances are likely to use many such paths for the first time and hence unexpected results may occur (Figure 1).
The European System and Software Initiative "Performance of Errors through experience-driven Test" (PET) investigated this phenomenon and agreed that deployed code often possesses many errors. Their findings demonstrated that the number of bugs that can be found by newer methods of testing, such as static and dynamic analysis, is quite large, even in code that has been through functional system testing and has subsequently been released.
It is often argued that legacy code which forms the basis of new developments has been adequately tested just by being deployed in the field. However, even in the field, it is highly likely that the circumstances required to exercise some parts of the code have never (and possibly can never) occur. It follows that many unexercised paths are likely to remain in software tested only through functional testing and in the field, and such applications have therefore sustained little more than an extension of functional system testing by their in-field use.
When there is a requirement for ongoing development of legacy code for later revisions or new applications, previously unexercised code paths are likely to be called into use by combinations of data never previously encountered (Figure 2).
Such situations may become particularly challenging when the legacy code has been developed by individuals who have long since left the development team, particularly if the legacy code originates from a time when documentation was not of the high standard expected today.
Given that commercial pressures often rule out a rewrite, what options are available?
Static Analysis of SOUP's Dynamic Behavior
Exercising all routes through legacy code may appear to be a daunting task, and there are a number of static analysis tools on the market which use mathematical techniques such as abstract interpretation to try to verify all possible executions of a program. As a result, they aim to prove which operations are free of run-time errors, including overflows, divisions by zero, buffer overflows, or pointer issues, and which identify where run-time errors will or might occur.
Such claims sound appealing, especially if engineers believe the half truth that problems can be isolated without any need for code to be changed or even understood.
It is indeed true that some problems,in simpler code sections can be readily isolated and corrected with relatively little knowledge of the code, and that such successes can provide great encouragement during tool evaluation. It is easy to derive some level of warm, fuzzy feelings that the software is reasonably robust in this way.
However, where source code is complex such tools have to rely ever more heavily on data approximations and hence raise many false warnings. These require the user to confirm or deny the existence of each problem in the very code sections which by definition are likely to represent the most complex parts of the application and the most obtuse parts of the sources.
Even setting that aside, these tools provide no evidence that the code is functionally correct -- and software which is proven to be robust but which remains functionally flawed is unusable.
Using Static and Dynamic Analysis In Combination
If the panacea of purely static techniques is a mirage, then how can the more traditional combination of static and dynamic methodologies be adapted to address the unique demands of SOUP?
The traditional application of formal test tools demands the sequence outlined in Figure 3.
The team facing the task of building on SOUP will be required to follow a more pragmatic approach not only because the code already exists, but also because the code itself may well be the only available detailed (or at least current) "design document". In other words, the existing code effectively defines the functionality of the system rather than any documentation. In enhancing the existing code, it is vital that the existing functionality is not unintentionally modified either by rewriting to meet coding standards or as the result of the enhancements under development.
The challenge is therefore to identify and use the building blocks within the test tools which, when used in a different sequence, can help in creating an efficient enhancement of SOUP.