In the foreword to High Assurance Design (by Cliff Berg; Addison-Wesley Professional, 2005), Neumann states that "Security and reliability are both emergent properties of an entire system..." A consequence of this is that many kinds of tests that must be performed to verify security and reliability are so-called "negative tests" that verify that something cannot be done, rather than verifying that something can be done. To do these kinds of testing, a complex testing configuration might be necessary that is best left in the hands of professional testers. As Brian Marick put it, "good programmers do functional testing, and testers should do everythingelse" (www.testing.com/writings/ classic/mistakes.html).
While considering a holistic system design, you must be careful to avoid falling into the pit of "big design up-front" (BDUF). You should be able to accomplish this if you follow agile principles regarding how the design is developed. Specifically, the criteria in Table 1 should be met for a design to be "agile."
|1. Create the minimal design that is necessary at the time.|
|2. Maintain only those parts of the design that need to be maintained to satisfy the mission of the applicationUpdate Only When It Hurts (see Agile Modeling).|
|3. Define success criteriateststhat the design must meet from the beginning, in a manner analogous to test-driven development (TDD), and require the design to be "tested" with each build of the system.|
As an example of item 3 in Table 1, consider an application for which the design includes a layer to protect an underlying resource, and for which the design includes a rule that "no module may directly access the resource except for the protection layer." Any code that violates this rule puts the entire protection scheme at risk. Therefore, verification of compliance with this rule must be achieved if it is to be successful. Because this rule cannot be tested by using functional tests, another means must be used. The testing of this design ruleits verificationcan be achieved through any or several means, including the methods in Table 2.
|1. Manual inspection of the entire code that might be able to directly access the resource.|
|2. Automated inspection, using code scanning or parsing tools.|
|3. Runtime checking, using dynamic tools.|
|4. Sufficient awareness on the part of developers, so that they know not to violate the rule.|
The choice should be made in consideration of the project's unique circumstances, including the level of maturity of the team, the size of the project, and the complexity of the overall code base.