Investigative Testing
As I was writing this column, I was lucky enough to attend a presentation given by Dr. Cem Kaner to the Toronto Association of Systems and Software Quality (TASSQ). Kaner, coauthor of Lessons Learned in Software Testing (Wiley, 2001), described his thoughts and experiences in software testing, and as a result, helped me to conceptualize some of my own ideas. In particular, I had been struggling to properly describe the activities of investigative testing efforts during construction iterations, and Kaner's presentation helped to coalesce my experiences.
A separate test team? Preposterous you say! Actually, there is significant value to be gained by submitting your system to an independent test team at intervals throughout the lifecycle so that they can verify the quality of your work. Agile teams produce working software at the end of each construction iteration; therefore, you have something new to test at that point. A common practice is to provide a new version of the system at least once a week, regardless of your iteration length, a particularly good strategy the closer you get to the End Game.
The investigative test team's goal should be to ask, "What could go wrong," and to explore potential scenarios that neither the development team nor business stakeholders may have considered. They're attempting to address the question, "Is this system any good?" and not, "Does this system fulfill the written specification?" The confirmatory testing efforts verify whether the system fulfills the intent, so simply repeating that work isn't going to add much value. Kaner promotes the idea that good testers look for defects that programmers missed, exploring the unique blind spots of the individual developers.
Investigative testers describe potential problems in the form of defect stories—the agile equivalent of a defect report. A defect story is treated as a form of requirement—it is estimated and prioritized and put on your requirements stack. The need to fix a defect is a type of requirement, so it makes perfect sense to address it just like any other requirement. As you would expect, during the End Game, the only requirement type that you're working on is defect stories.
Your investigative testing will address common issues such as load/stress testing, integration testing, and security testing. Scenario testing, against both the system itself and the supporting documentation, is also common. You may also do some form of usability testing—the user interface is the system to most end users; therefore, usability is critical to success. The UI includes both the screens that people interact with and the documentation that they read, implying that you need to test both.
Good investigative testing efforts reveal any problems that developers missed long before they become too expensive to address. It also provides feedback to management that the team is successfully delivering high-quality working software on a regular basis. Kaner pointed out that there's no one right way to go about investigative testing, nor is there one correct list of techniques to employ. Your efforts must reflect the goals of the project team that you're supporting. For example, is the goal to determine whether the system is ready to be shipped? Is it to ensure that the system interoperates with other existing systems? Is it to help the developers to identify problems in their own testing efforts by pointing out the causes of defects that they missed? Is it to minimize the chance of a lawsuit against your organization or its managers? The context in which you are testing will determine what and how you test—not only will the context be different for each project; the context also changes over the life of the project.
The type of confirmatory testing performed by agile teams is only one part of the testing picture—it is the agile equivalent of traditional smoke testing. This is a great start, and having automated regression testing provides the safety net required by evolutionary development techniques. Investigative testing enables you to explore the critical "big picture" issues, as well as the "little picture" issues that nobody thought of until now, which confirmatory testing typically does not.
