How Can I Test Java Classes?

Krishnan is president of Man Machine Systems and can be contacted at [email protected] or http://www .mmsindia.com.

Unit testing is an important element of testing software. In object-oriented software development, this term commonly refers to testing individual classes. The term "integration testing" denotes testing a group of classes, while the term "system testing" refers to testing the application as a whole. In the simplest and most widely followed testing practice, developers of classes are expected to perform unit and integration testing of their respective classes before releasing their library to other groups. The test team is responsible for system testing before releasing the application to clients.

There is on-going research in the area of testing object-oriented software, and a lot more needs to be done. Because class testing is the starting point of testing object-oriented software, understanding the issues involved and devising a way to perform effective class testing is a crucial first step in delivering high quality software. At Man Machine Systems, we have developed a new model for testing Java classes, called "invasive testing." We believe this model can make class testing more interesting and effective. Testing is typically characterized as a destructive process, so the term effectively suggests that the tester must be able to uncover a maximum number of bugs with minimal effort. (The term "tester" as used here denotes a role rather than an individual. This means that a developer could be a tester.) In this article, I'll describe the invasive testing model and compare it with conventional unit-testing techniques.

Invasive and Intrusive Testing of a Class

Invasive testing is a technique that uses the normally inaccessible elements of a class to increase the effectiveness of testing. When the class under test is modified to support testing, it is called "intrusive testing."

If a class C comprises a set M of methods that belong in the class interface, intrusive testing typically assumes the existence of N additional methods and modifications to some or all of the M methods to aid testing. The additional N methods are not intended to be used by normal clients of the class (and are not part of the published class interface), but are defined purely to support testing. If methods in sets M or N access private members of the class, then the technique is invasive as well. Plain invasive testing does not modify the class under test, but would use private details of the class that normal users cannot access.

Conventional Class Testing Strategies

There are a number of conventional class testing strategies, including those that are noninvasive/nonintrusive, noninvasive/intrusive, and invasive/intrusive. Table 1 compares the different techniques I discuss here.

The noninvasive/nonintrusive tester-as-client is the most common strategy for testing classes. Here, the tester writes a driver in the same language as the class under test, and exercises the published methods of the class, following a set of test criteria, such as coverage. Since the driver code resides in a different class than the tested one, accessibility rules defined in the language apply. Consider the stack implementation MyStack in Listing One as an example of a Java class to be tested. The test driver in Listing Two tests the stack class. Assume the existence of test support routines such as Test.error(String). The test driver is written to test invocations of MyStack's public methods in some combination.

One benefit of this technique is that it is intuitive, because it simulates actual usage of the class under test. Also, the class tested is the class actually released to the customer because no modification is performed to the class for testing purposes.

This approach does have its limitations, however. Since the tester has to exercise public methods in the class interface, certain methods can only be tested in combinations, not individually. For example, the push (or pop) method cannot be tested in isolation. As a result, if the push/pop combination fails, it could be due to a bug in either or both the methods, and additional tests may be needed before the actual bug may be isolated. Even if the push/pop combination matches, there is no guarantee that they are both correct; it is conceivable that both have bugs that coincidentally produce the correct behavior. Once again, additional combinations will have to be tried. Also, it is not always possible to exercise all parts of the implementation adequately by invoking public methods alone. Finally, there is no way to test interfaces.

In the noninvasive/intrusive tester-as-client scenario, the class to be tested is modified to support testing. Accessible getter and setter methods are added for all state variables that are otherwise inaccessible. Alternatively, all elements of the class are made public to grant unrestricted access. This permits the tester to overcome the limitations of the noninvasive/nonintrusive approach. The MyStack class in Listing One, according to this strategy, might be modified as in Listing Three.

The test driver is now rewritten to take advantage of private details of the class to perform more effective testing. Listing Four shows how this can be done.

One benefit of this approach is that the semantics of individual methods can be verified. This can reduce the test effort, because the number of methods is much less than the number of method combinations.

The limitations of the technique, however, are that the class tested is different than the one released to a client, and that the tester needs to understand the class implementation details, in addition to its interface. For complex classes, this could be quite difficult. Also, if the class implementation changes in the future, the driver code will have to be appropriately modified. Finally, the source code for the class is required.

An invasive/intrusive technique, which is to automatically instrument the code with assertions, is a promising approach for testing object-oriented software. In this case, the developer embeds assertions (class invariant, method preconditions, and method postconditions) inside comments in the source code. A preprocessor parses these assertions and emits the modified source where the assertions have been appropriately moved to method bodies. The preprocessed code is compiled using a standard Java compiler. The resulting application is then run as if it were the original program. Any violation of contracts will be reported by the assertions layer. Listing Five is an example of how assertions might be specified.

The assertion language depends on the tool used; currently there is no standard. The previous example uses a syntax supported by our tool, JMSAssert, which generates a JMScript test script file. Reto Kramer's iContract (http://www.reliable-systems .com/) is a tool in the public domain that supports assertions in Java classes.

One benefit of this invasive/intrusive technique is that reasonably complex class invariants, as well as preconditions and postconditions, may be specified. Since the assertions are defined as part of class/method comments, the source can be recompiled to get uninstrumented classes. Finally, this approach assists in testing the complete application.

However, one of its limitations is that the class tested is different than the one released to a client. Also, source code has to be available to instrument with assertions. Furthermore, in many cases, the assertion language is proprietary, so moving to another preprocessing tool is difficult and to test classes individually (not as a sealed-off application), a test driver still has to be written. Finally, individual methods cannot be tested because of access restrictions.

Another invasive/intrusive approach is when the tester is the developer. In this case, the test driver resides within the class being tested; see Listing Six. Typically, this is the main method or a test method that is static. When it is invoked, the test logic is exercised. Because the method is part of the class, access restrictions do not apply; all elements of the class can be accessed.

One of the benefits of this invasive/intrusive approach is that there is no need to convert private elements to public for the sake of testing. Also, the test driver for the class is self-contained, so the driver is more likely to be in sync with the class.

Its limitations are that the test driver cannot be reused when the class evolves. For each version of the class, a corresponding version of the driver inside the class (perhaps copying and pasting from a previous version) must be developed. To facilitate regression testing, it is better to have the driver separated from the class. Again, the source code must be available, and when the class is delivered to a client, the test driver constitutes excess baggage. However, if the driver is removed from the code, then two versions of the class have to be maintained, one with the driver and another without the driver.

Invasive Testing

It is a widely acknowledged fact that testing object-oriented software is more difficult than testing nonobject-oriented software. In "Design-for-Testability For Object-Oriented Software" (Object Magazine, July 1997), Jeffery E.Payne, Roger T.Alexander, and Charles D.Hutchinson, point out that "from the value of testing perspective, information hiding reduces the ability for faults to propagate to an observable output and hence reduces the likelihood that faults will be revealed during testing."

The invasive model lets the tester tunnel through the information-hiding barrier and write a driver that accesses the normally inaccessible elements of a class. This can occur without making changes to the class source. In fact, the approach does not require Java source code to be available for testing purposes. To assist in achieving this, we've developed a scripting language called "JMScript" that is layered on Java. The language lets testers write a script that instantiates Java objects and send messages to them. This is similar to what happens in Java itself. Like many other scripting languages, JMScript is weakly and dynamically typed. These types are associated with values of objects and not with variables. Once a Java object is instantiated in the script, any element of the object, be it private or protected, can be accessed without restriction.

The JMScript driver (available electronically; see "Resource Center," page 5) illustrates one way of testing the MyStack class. The script can be executed by running the command-line version of the interpreter (or the more sophisticated JVerify environment can be used, that includes the interpreter and much more).

The script accesses even the private elements of a class, bypassing Java's access control mechanism. Notice how assertions are used after every method call to check the postcondition. However, explicitly checking the postconditions can litter the script with too many assertions. There is a more convenient way to do this, as in the test script in MyStackTest2.jms (available electronically), which illustrates that a tester can identify an invariant for a class, a set of preconditions and postconditions for methods and register those with the interpreter. When a method is invoked from the test script, the corresponding precondition, postcondition, and invariant routines are automatically and appropriately executed by the interpreter. The sequence followed by the interpreter is:

• Check class invariant.
• Check precondition for the method.

• Invoke method.

• Check postcondition for the method.

• Check class invariant.

There are exceptions to this rule, but that is beyond the scope of this article.

It is possible to do better than setting up triggers manually. If the class source is available and assertions are embedded in comments as in Listing Five, another utility called "JMSAssert" will parse these and automatically generate a script file containing appropriate triggers.

Testing State Machines

The invasive model provides a convenient approach to testing state machines. In general, testing a state machine tends to be fairly involved because of the potentially infinite number of states possible. The ability to access private details could considerably reduce the effort (the downside is that intimate knowledge of the class may be required).

Consider the state machine in Figure 1. (The Java source code for the state machine is available electronically. For clarity, I've left out several optimizations.) How can you prove that the Java state-machine program accepts all the strings as per the state-machine specification of Figure 1? Clearly, testing with all strings is impossible. Fsmtest.jms (available electronically) shows how the state machine can be tested using JMScript. The basis for the testing strategy just used is to drive the state machine to each intermediate state not allowed normally, and then to ensure that when an input is received, it transitions correctly to the next state.

Invasive testing's benefits include the fact that the class under test is not modified to support testing. The class tested is the same as the class released to a client. The source code for the class does not need to be available. (This means it is possible to test third-party libraries.) The test driver accesses implementation details, if necessary, to test individual methods, not combinations. This can minimize testing effort. Class invariant, method preconditions and postconditions can be complex in real situations. The support for such arbitrarily complex conditions is taken care of by providing procedures in JMScript. Because the test driver is separate from the class being tested, reuse of the test cases is possible throughout regression testing.

A limitation is that the test script is written in a language different than Java. This means learning a new language. This approach is ideally suited to perform unit testing of individual classes and integration testing of small groups of classes, but not for system testing. This relies heavily on the commitment of the development environment to the use of "Design by Contract."

JVerify Test Environment

To create, execute, and test JMScript scripts, we developed JVerify, a GUI-based Windows application. The environment supports a debugger that allows setting breakpoints, and examining the execution snapshot through an object browser and call stack. Another salient feature of the environment is the facility to understand a class's interface from its .class file. This can be useful when the source for the class under test is not available.

Conclusion

Testing object-oriented software is more difficult because of information hiding. Invasive testing makes a tester all powerful by allowing access to private details of the class under test. This model renders a class more amenable to testing (even if it has not been designed for testability). This model is also particularly useful for testing state machines. JMScript, layered on Java, has been designed to support class invasion in the context of Java programs. More information about the language and associated tools is available at http://www .mmsindia.com/.

Acknowledgments

The JVerify project would not have been possible without Krishnan and Sunil. I thank them for months of hard work. I am grateful to Steve Brothers of CyberPlus Corp. and Ashok of Verifone for insightful comments on the draft.

DDJ

Listing One

class MyStack {
   private Object[] elems; 
   private int top, max; 
   public MyStack(int sz) {
      max = sz;
      elems = new Object[sz];
   }
   public void push(Object obj) throws Exception {
      if( top < max )
         elems[top++] = obj;
      else throw new Exception("Stack overflow");
   }
   public Object pop() throws Exception {
      if( top > 0 )
         return elems[--top];
      throw new Exception("Stack underflow");
   }
   public boolean isFull() {
      return top == max;
   }
   public boolean isEmpty() {
      return top == 0;
   }
}

Back to Article

Listing Two

class StackTester {
   public static void main(String[] args) {
      MyStack s1 = new MyStack (10); // Max of 10 elements
      if( !s1.isEmpty() )
        Test.error("Stack is not empty initially!");
      StackTester obj1 = new StackTester(); // An object to stack
      s1.push(obj1);
      if( s1.isEmpty() )
        Test.error("Stack is empty after a push!");
      StackTester obj2 = (StackTester) s1.pop();
      if( obj1 != obj2 )
        Test.error("Problem in push()/pop()!"); // ----- (A)
           // ... Other code
      }
}

Back to Article

Listing Three

class MyStack {
    // Private elements are public - FOR TESTING ONLY
    public Object[] elems; 
    // Private elements are public - FOR TESTING ONLY
    public int top, max; 
    public MyStack(int sz) {
       max = sz;
       elems = new Object[sz];
    }
public void push(Object obj) throws Exception {
       if( top < max )
          elems[top++] = obj;
    else throw new Exception("Stack overflow");
    }
       public Object pop() throws Exception {
       if( top > 0 )
          return elems[--top];
       throw new Exception("Stack underflow");
    }
    public boolean isFull() {
       return top == max;
    }
    public boolean isEmpty() {
       return top == 0;
    }
}

Back to Article

Listing Four

class StackTester {
  public static void main(String[] args) {
    MyStack s1 = new MyStack (10); // Max of 10 elements
    if( !s1.isEmpty() ) 
    Test.error( "Stack is not empty initially!");
    StackTester obj1 = new StackTester(); // An object to stack
    s1.push(obj1);
    Test.assert((s1.top == 1) && (s1.elems[s1.top-1] == obj1),"Push failed!");
    StackTester obj2 = (StackTester) s1.pop();
    Test.assert( (s1.top == 0) && (obj2 == obj1), "Pop failed!");
    // ... Other code
    }
}

Back to Article

Listing Five

/** @inv (top >= 0) && (top < max) */
class MyStack {
   private Object[] elems; 
   private int top, max; 
   /** @post (max == sz) && (top == 0) */
   public MyStack(int sz) {
      max = sz;
      elems = new Object[sz];
   }
   /** @pre top < max; 
   *   @post (top == top$prev + 1) && (elems[top-1] == obj)
   */
   public void push(Object obj) throws Exception {
     if( top < max )
        elems[top++] = obj;
      else throw new Exception("Stack overflow");
   }
   /**  <assertions here> */
   public Object pop() throws Exception {
      if( top > 0 )
         return elems[--top];
      throw new Exception("Stack underflow");
   }
   public boolean isFull() {
      return top == max;
   }
   public boolean isEmpty() {
      return top == 0;
  }
}

Back to Article

Listing Six

class MyStack {
   // ... Insert stack related elements here.
   // ...
   // This is the stack test driver. It is part of the class itself.
   public static void main(String[] args) {
      MyStack s1 = new MyStack (10); // Max of 10 elements
      if( !s1.isEmpty() ) 
         Test.error( "Stack is not empty initially!");
      MyStack obj1 = new MyStack (); // An object to stack 
      s1.push(obj1);
      Test.assert( (s1.top == 1) && (s1.elems[s1.top-1] == obj1), 
                                                           "Push failed!");
      MyStack obj2 = (MyStack) s1.pop();
      Test.assert( (s1.top == 0) && (obj2 == obj1), "Pop failed!");
      // ... Other code
   }
}

Back to Article