Continuous Integration and Performance Testing

Performance, Integration, And Load Tests

Now you can add performance tests into the continuous integration process. The most straightforward method is to create a new virtual project in CruiseControl, one that uses the same build script but executes a performance unit-testing target. The following shows the key changes you could make to a copy of Listing One's project definition:

 
<project name="ant-junit-performance-unit-tests">
 ...
  <schedule interval="600">
    <ant anthome="apache-ant-1.6.5"
       buildfile="projects/${project.name}/build.xml"
           target="performance-unit-tests.execute" />
  </schedule>
  <publishers>
   <onsuccess>
      <artifactspublisher dest="artifacts/${project.name}"
         dir="projects/ant-junit/profiling-reports"/>
    </onsuccess>
  ...

In this case, CruiseControl is configured to check for source updates every five minutes—because performance tests take longer to run, they cannot and should not be run as frequently. In a real-world application, checking hourly is more reasonable.

But most importantly, a new Ant task, performance-unit-tests.execute, is run, and the results are saved to a profiling-reports directory. The actual definition of this Ant task and the performance profiling process is entirely dependent on the analysis tool used for this. Besides commercial code analysis tools, some promising looking open-source alternatives are emerging, such as EMMA (emma.sourceforge.net).

Similarly, you can perform integration and load testing in this continuous integration environment. The freely available Apache JMeter is my choice for a simple load generator for our web-based application. It's simple enough to integrate JMeter with Ant using an Ant task created by Programmer Planet (programmerplanet.org/media/ant-jmeter/ant-jmeter.jar).

Once integrated, you define a JMeter Ant task by adding the following to the Ant script:

<taskdef name="jmeter"
classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>

JMeter test plans are written in XML and they specify the details of the load test. These test plans are called by the JMeter Ant task, which outputs the results to a log file. JMeter test plans can define such metrics as the number of virtual users (threads) and test iterations (loops). You can configure the JMeter Ant task to run different test plans to test basic integration and also to perform full load testing.

Although JMeter outputs a comma-separated values (CSV) file by default, you can change this to XML using the jmeter.save.saveservice.output_format=xml argument. This lets us programmatically determine success or failure, and produce a human-readable report using an XML style sheet (XSL). This report can be evaluated against the application's service-level agreements to give an idea, during development, of whether it meets its scalability requirements. It is possible to further integrate cross-JVM performance analysis tools to help diagnose scalability issues.

Conclusion

Continuous integration is becoming increasingly popular as teams see the benefits of automating more aspects of the development process. Extending continuous integration to different types of performance testing makes the investment in continuous integration even more worthwhile.