Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Channels ▼
RSS

JVM Languages

The JUnit++ Testing Tool


Feb01: The JUnit++ Testing Tool

Java integration, system, load, and stress testing

Siegfried is a technical architect for Software Daten Service. He can be contacted at [email protected].


JUnit is a freely available Java unit test framework created by Erich Gamma and Kent Beck. Currently at Version 3.2, JUnit is extensively used in the Java community because of its elegance of design and ease of use. Over time, JUnit (available at http://members.pingnet.ch/gamma/junit.htm) has been extended to address issues related to integration, system, load, and stress testing. However, these extensions have inevitably resulted in shortcomings such as the lack of a test data repository, which stems from the original design. The absence of a test data repository results in hard-coded test data that increases the maintenance costs of testing database-driven systems. This should not be underestimated since the test code should have the same quality requirements as the application code. It is common to ship the regression test suite as part of the deliverables to increase customer confidence and supplement the documentation. To address this and other limitations, we've extended the JUnit test framework to provide a test data repository, command-line arguments, and an improved TestRunner class that supports a built-in repetition counter and multithreading on the command line. This improved version, called "JUnit++" (available electronically; see "Resource Center," page 5), has been used for more than a year to test an online banking application — currently at more than 100,000 lines of code — at Software Daten Service (Vienna, Austria).

Using the JUnit Extensions

The main improvement of JUnit++ is the provision of a test data repository accessed by subclassing junit.extensions.ConfigurableTestCase. Listing One illustrates how you can use the test data repository for a simple test. The class ConfigurableTestCase uses java.util.Properties to load the test data. The loading of the property file is triggered in the constructor of ConfigurableTestCase. For a master test suite that only invokes other test suites, this strategy doesn't work because no constructor is invoked. In this case, loading the property file has to be done manually by invoking initTestProperties() in the suite() method; see Listing Two, which shows how to use the test data repository for a master test suite.

How is the property file found when an instance of ConfigurableTestCase is executed? It is assumed that the property file has the same name as the class file but a different extension; that is, the property file for the FooTest.class would be named FooTest.ini. The property file is either looked up in the current directory or in the list of directories defined in the classpath variable.

In some projects this approach might not work (for instance, using a single property file for all test suites); therefore, the system property junit.data either defines a property file or a starting directory for the search; see Example 1(a). How is an entry in the property file defined? To access the test data repository and the class name, the name of the test case and the property name are concatenated to generate the key in the following order:

1. Fully qualified class name plus test name plus property name.

2. Class name without package name plus test name plus property name.

3. Class name without package name plus property name.

4. Property name.

This implementation allows the reuse of test data definitions shared by one or more test suites. For example, Example 1(b) shows the name of the server used for testing a client/server application with multiple test suites. In the case of a master test suite, the corresponding property file contains references to the property files of the contained test suites, as in Example 1(c), which are loaded recursively.

What happens if a different type of test data repository already exists (as in a relational database)? In JUnit++, the implementation of a test data repository is determined and instantiated at run time. Hence, it is possible to provide a different implementation of a test data repository by defining the implementation class at the command line, as in Example 2(a). After a successful test execution, you might reuse the test suite to gather performance data either by simply measuring the execution time or profiling the application. The profiling gives a good indication of how much time and resources are spent in an ORB run-time library or a Servlet engine. It is necessary to fine tune the number of test executions to find a balance between the effect of one-time initializations distorting the result and time required to generate the profiling data. By using junit.extensions.TestRunnerEx, the number of test repetitions can be defined on the command line in Example 2(b).

If the profiling of a client/server application is done on one computer, it will become overloaded. In this case, a delay of the test case invocations can be defined on the command line in Example 2(c). The next step might be overloading the server until it crashes due to race conditions and/or resource shortage. This can be accomplished by invoking one or more test suites using multiple threads, repetition counters, and test case invocation delays to simulate a more realistic user behavior; see Example 2(d).

Some components require command-line parameters for initialization. JUnit++ provides a uniform way of defining application-specific command-line parameters by passing them as a system property junit.argv. Some development environments are unable to process quoted arguments when invoking the debugger. This is why multiple arguments can be separated by the pipe character, as in Example 2(e).

Sometimes passing arguments via command line is clumsy, for example, when initializing a JDBC connection for multiple test suites. In this case, a ConfigurableTestCase can be used (see Listing Three), where the parameters are stored in a corresponding property file.

If a test suite does a lot of printing, it is useful to get a more verbose output of a test run. This can be accomplished by turning on the verbose mode of TestRunnerEx, as in Example 2(f).

Testing an Online Banking Application

The online banking application is a satellite system of the Global Entity Ordering System (GEOS), a host-based application consisting of 6 million lines of code written in C. It provides multiple front ends (Swing, HTML, WAP) and is implemented in Java utilizing CORBA, servlets, and XML. Figure 1 illustrates the application's architecture.

The GUI framework is based on the JWAM architecture (see "Frameworkbasierte Anwendungsentwicklung," by Guido Gryczan, Carola Lilienthal, Martin Lippert, Stefan Roock, Henning Wolf, and Heinz Züllighofen, OBJEKT Spektrum, January 1999) and generates requests which are delegated to a Datasource. The Datasource, in turn, delegates the requests to the Transport Layer, which shields the client from the underlying middleware. The requests are transmitted to the CORBA-based Business Server implemented by a CorbaBusinessObject, which maps the requests to various Data Access Objects. The CorbaBusinessObject and SecurityManager are part of an application framework developed in-house. The Data Access Object is ultimately responsible for contacting the host-based GEOS, which uses a relational database. The results are propagated back through the various layers until they are displayed by the front end.

Using a bottom-up test strategy, the architectural layers are tested with the help of JUnit++ in reverse order, starting with the Data Access Objects, the CORBA application framework, and the Datasources.

The regression tests for the Data Access Objects are organized by means of a hierarchy of test suites. Each test suite is derived from ConfigurableTestCase and retrieves the test data from a single property file for each database. By simply defining a different property file on the command line, the test suites can be run using a different database.

The CORBA application framework regression test consists of 11 individual test suites that are invoked through a master test suite. The test suites are not only used for regular regression testing but also for stress testing the ORB and optimizing the performance.

Two Examples

The implementation of a data conversion module copying internal data structures to IDL data types was considered too slow — 10 iterations of the original regression test suite took 32 seconds to execute. By optimizing the code using a Java profiler, we were able to reduce the execution time significantly:

  • Accessing the metadata of the internal data structure was accidentally done with a function that didn't use a metadata cache. By using cached metadata, the execution time was reduced to 22 seconds.
  • Somewhere in the code, a linked list was used but the access was done by index. Changing the implementation to a java.util.ArrayList reduced the execution time to 14 seconds.

  • A handcrafted function for creating a timestamp instead of using the Java API reduced the execution time to 10 seconds.

  • Some optimizations within the actual data transformation code reduced the execution time to 8 seconds.

The regression test for the Datasources also acts as an integration and system test because this test exercises multiple layers of the application — the transport layer, middleware, business server, and Data Access Objects. By using JUnit++, this test can be easily turned into a stress test executed over the weekend.

In a second series of tests, we used a test configuration to simulate the load of 35,000 clients within nine hours so that all server processes and infrastructure components failed. A closer inspection revealed the following faults:

  • Some tests intentionally didn't close their sessions to test the distributed session garbage collection triggered by a session timeout. These sessions never timed out due to a coding error.
  • The high load resulted in thrashing of the server processes and socket timeouts. The default behavior of the ORB in case of a socket error is a fail-over to a replica server, which increased the load and the number of allocated threads further.

  • The high number of sessions, connections, and threads lead to a shortage of virtual memory, which crashed the CORBA Naming Service.

Having multiple regression tests, it is reasonable to use the test suites in order to create a Daily Build and Smoke Test (see Rapid Development, by Steve McConnell, Microsoft Press 1996). This guarantees a certain level of quality because the regression tests cover a large portion of the functionality and span multiple layers of software.

Conclusion

JUnit++ makes it possible to separate test data from test code, which is essential for testing database-driven applications. Omitting this separation results in hard-coded test data, which increases the tests maintenance costs and decreases the acceptance of regression tests.

The ease of generating an arbitrary load under various conditions encourages developers to use stress tests for components early in development, which in turn, increases the reliability of the final product. This observation is particularly relevant for distributed systems because it might require an expert's knowledge to trace a failure back to a single component.

DDJ

Listing One

public class FooTest extends ConfigurableTestCase {
    public FooTest ( String name ) {
        super(name);
    }
    public void testFoo() {
        String stringValue   = getString( "key1" );
        Boolean booleanValue = getBoolean( "key2" );
        Integer integerValue = getInteger( "key3" );
        Double doubleValue   = getDouble( "key4" );
    }
}

Back to Article

Listing Two

public class AllTests extends ConfigurableTestCase {
    public AllTests ( String name ) {
        super(name);
    }
    public static Test suite() {
        // load property file
        initTestProperties( AllTests.class );

        TestSuite suite= new TestSuite();
        suite.addTest( FooTest.class );
        suite.addTest( BarTest.class );
        return suite;
    }
}

Back to Article

Listing Three

public class JDBCTestSetup extends ConfigurableTestSetup {
    static private Connection connection = null;
    public JDBCTestSetup () {}
    public void setUp() {
       connection = DriverManager.getConnection(
            getString( "url" ), 
            getString( "user" ),
            getString( "password" ); 
    }
    public void tearDown() {
        connection.close();
    }
    public Connection getConnection() {
        return connection;
    }
  }
}


Back to Article


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.