Enhancing SOUP With a Modern Test Tool Suite
To decide on a more pragmatic approach, it is necessary to answer some fundamental questions:
- What level of understanding of the existing code is available to the development team?
- Does the existing code need to be upgraded to meet any new standards not previous enforced?
- How can the sections of code affected by changes be tested? Can code coverage levels be proven to have adequately exercised that code?
- What if the nature of the code is such that modularity is compromised? How can new sections of code then be proven?
- How can the revised code be proven to be functionally equivalent to its SOUP origins?
Improving the Level of Understanding
The system visualization facilities provided by modern test tools is extremely powerful, whether applied in terms of statement blocks, procedures (or classes), applications and/or system wide. Static call graphs provide a hierarchical illustration of the application and system entities, and static flow graphs show the control flow across program blocks, as in Figure 4. The use of such color-coded diagrams can be of considerable benefit when understanding SOUP.
Such call graphs and flow graphs are just part of the benefit of the comprehensive analysis of all parameters and data objects used in the code. They are complemented by utilities such as automatic header comment generation, and data flow reporting to provide details of the relationships between the component parts of the software and problems associated with those relationships. Such information is particularly vital to enable the affected procedures and data structures to be isolated and understood when work begins on enhancing functionality.
Enforcing New Standards
When new developments are based on existing SOUP, it is likely that internal, industry, or international standards will have been enhanced in the intervening period. Code review analysis can highlight any contravening code for correction.
It may be that the enforcement of a full set of current coding rules to SOUP is too onerous and so a subset compromise is preferred. In that case, it is possible to apply a user-defined set of rules as in Figure 5, which allows such a subset to retain cross referencing to the international standard.
Where legacy code is subject to continuous development, it is likely that production pressures generally outweigh the longer term ideal to adhere to more demanding standards. It may be pragmatic to initially establish a relatively lenient set of coding rules, to isolate only the most unwanted violations. A progressive transition to a higher ideal may then be made by periodically adding more rules with each new release, so that the impact on incremental functionality improvements is kept to a minimum.
Test tools cannot correct the code. However, they enable the correction of code to adhere to such rules as efficiently as possible. Using a "drill down" approach, the test tools provide a direct link between the description of a violation in a Code Review Report and an editor opened to the line of code which includes that violation.
Ensuring Adequate Code Coverage
Again, testing based on a statistical sample of a program's expected input data can lead to testing being focused on a particular value or range of values. Code proven in service has effectively been subjected only to similar, if extensive, functional testing which has serious implications for SOUP enhancement.
Structural Coverage addresses this issue by testing equally across the data, assuming each value has an equal chance of occurring.
Although system-wide functional testing will not test all paths, clearly it will test many. Given that functional software exists at the outset, it provides a logical place to start. Test tools can provide the means to identify which parts of the software have been exercised, and so highlight those areas still requiring attention.
The test tool takes a copy of the code under test and implants additional function calls ("instrumentation") to identify the paths exercised during execution. Using this "white box" testing technique, tools provide statistics to show how much of the code has been used. Coloured call graphs and flow graphs complement reports to give an insight into the code tested, and to clearly show the nature of data required to ensure additional coverage.
System-wide testing inevitably leaves some paths unproven. However, because code instrumentation can be applied on a unit test or system wide basis, it is possible to devise unit tests to exercise those parts of the code which have yet to be proven through system test. This is equally true of code which is inaccessible under normal circumstances, such as exception handlers and defensive coding.
Unit testing can be used to ensure that each section (or unit) of the code functions correctly in isolation. Whether testing a single function/procedure or a subsystem within an application, the time involved in manually constructing a harness to allow the code to compile can be considerable and can demand a high level of expertise on the part of the tester.
Modern unit test tools minimize that overhead by automatically constructing the harness code within an easy-to-use GUI environment and providing details of the input and output data variables to which the user may assign values. Such tools can overcome traditional headaches such as providing access to C++ private member variables for test purposes. The result can then be exercised on either the host or target machine. Sequences of these test cases can be stored, and they can be automatically exercised regularly (perhaps overnight) to ensure that ongoing development does not adversely affect proven functionality.
Dealing With Compromised Modularity
In some SOUP applications, terminology such as "Unit test" and "subsystem" do not readily spring to mind. Structure and modularity often suffer in such code bases, which does not make them any less vital to the organization but can challenge the notion of testing functional or structural subsections of that code. However, unit test tools can be very flexible in these matters, and the harness code which is constructed to drive test cases can include as much or as little of the source code base as is deemed necessary by the user. If it is necessary to include a large section of it to test the behaviour of particular functions within a system, then so be it.
The ability to do that may be sufficient to suit a purpose: "letting sleeping dogs lie." However, if a longer term goal exists to improve matters, then using instrumented code can be a vital tool in helping to understand which execution paths are taken when different input parameters are passed into a function -- either in isolation or in the broader context of its calling tree.
Ensuring Correct Functionality
From a pragmatic viewpoint, perhaps the most important aspect of SOUP-based development is ensuring that all aspects of the software functions as expected, despite changes to the code and to the data handled by it. In making changes to the code either to meet standards or to add functionality, it is therefore of paramount importance that functionality is not inadvertently changed.
Of course, unit tests could be used throughout to ensure that is the case but even with the aid of test tools, that may involve more work than the budget will accommodate. However, the concern here is not checking that each function is working in a particular way; more that however it works beforehand, it is not changing, except when there are deliberate attempts to make it do so.
Automatic test case generation is key here. By statically analyzing the code, test tools can automatically generate test cases to exercise a high percentage of the paths through it. Input and output data to exercise the functions is generated automatically, and then retained for future use.
They can then be used to perform batch regression testing on an ongoing basis, perhaps overnight or weekly, to ensure that when those same tests are run on the code under development there are no unexpected changes recorded. These regression tests automatically provide the cross reference back to the functionality of the original source code.
This ensures that when the enhanced software is released, clients can benefit from the new functionality and improved robustness of the revised code without fear of nasty surprises!
In an ideal world, test tools should be applied from the beginning of a structured and formal development process. However, sometimes commercial realities mean that legacy SOUP code is used as a basis for further development. By using those same test tools in a pragmatic way in these circumstances, it is possible to develop legacy SOUP code into a sound set of sources which are proven to be fit for purpose -- all in an efficient and cost effective manner.