|
Steven extends continuous integration to perform integration, performance, and load testing.
February 07, 2008
URL:http://www.drdobbs.com/jvm/continuous-integration-and-performance-t/206105978
Steven is the Java EE Domain Expert at Quest Software. He can be contacted at [email protected].
Automation has arguably done more to improve the development lifecycle in recent years than virtually anything else. As applications have become increasingly complex, iterative, and distributed, we seek to automate the software development process even further, beyond such mainstream development practices as automated source control and nightly build systems, towards automating integration and testing.
Continuous integration is one such "next step," one that can be extended to also automate certain kinds of testing. In this article, I extend continuous integration systems to perform integration, performance, and load testing. While this discussion uses Java EE and associated tools, the concepts are equally applicable to other platforms.
Continuous integration is a software engineering process where an application under development is completely rebuilt and tested frequently and automatically. While not an entirely new concept, the term was coined by Martin Fowler in a paper (martinfowler.com/articles/continuousIntegration.html) and grew in popularity as part of extreme programming (XP) practices and methodologies.
Many of us have horror stories of working on large development teams, where we each worked independently on our portions of an application, then spent as long (or longer) integrating all the pieces together into a working application. Part of the appeal of XP is the notion of integrating the application continually when new code is checked in, or at least several times a day.
A continuous integration process requires (paraphrased from Fowler's paper):
Continuous integration processes go beyond simply ensuring that the automating build compiles cleanly. To declare a build to be successful, it must pass basic functional tests. So we do need a somewhat mature development process, where teams all work from the same code repository, write a test case for the functionality they develop, build and test their functionality locally, refactor, and verify—only then committing their changes (frequently, at least daily) to the source code repository.
The continuous integration server includes functionality that detects that code has changed, checks out the code, and then builds and tests the entire application in a clean environment, eliminating the vagaries of individual developer's environments. If the build or tests fail, developers are notified immediately, when they are most able to easily and quickly fix the problem they introduced.
That's the process, simplified. For large applications, we often need to set up a staged build process, which lets us reduce the time needed to build and test (testing is usually what takes the most time).
The automated testing currently done in continuous integration environments is along the lines of functional or unit tests; that is, they verify that the application seems to do what is intended. Can you extend this testing to verify that the application seems to conform to performance, memory usage, integration, and scalability forecasts? It turns out that you can—I call the resulting process "continuous performance management."
Continuous performance management implements performance and scalability testing within a continuous integration environment. The idea is to configure what Fowler calls "secondary continuous integration builds" to execute performance unit tests, integration tests, or load (stress) tests.
This lets you, for example, profile the application's unit tests to identify slow algorithms, incorrect memory usage (such as loitering objects and object cycling), and measure code coverage of tests. You can also test beyond the component level, such as testing integration of components into a working solution by tracing a request as it passes between multiple Java Virtual Machines (JVMs). You can even run automated load tests within the continuous integration test harness to baseline and track the application's scalability during development.
The first prerequisite is having unit tests for your code developed and implemented in a scriptable framework such as JUnit (www.junit.org). This covers component-level unit-test analysis. Integration and load testing requires a different test bed because these are more like business use cases rather than functional tests—the HttpUnit extension of JUnit works well here.
To take fullest advantage of continuous performance management, you also need scriptable performance analysis tools; for example, a Java code profiler and memory debugger. That is, you need an engine that will run the performance, integration, or scalability tests, and capture the results.
So let's look at a real working continuous integration environment that has been extended to performance testing, the one I use daily in my work.
There are various continuous integration servers available, but for this exercise I chose CruiseControl (cruisecontrol.sourceforge.net). After downloading and installing the Windows version, the first step is to check out a copy of the application's source code to the project's directory on the continuous integration server. The following checks out the "trunk" branch of my "tddci" Subversion repository, which I've set up for this exercise with a single project named "ant-junit":
svn checkout file:///c:/lib/svn-win32-1.4.3/repositories/tddci/trunk
CruiseControl's central config.xml configuration file specifies the details of the projects it is responsible for building. Listing One is the initial CruiseControl project definition for continuous integration of our ant-junit project.
<project name="ant-junit">
<listeners>
<currentbuildstatuslistener
file="logs/${project.name}/status.txt"/>
</listeners>
<bootstrappers>
<svnbootstrapper localWorkingCopy="projects/${project.name}" />
</bootstrappers>
<modificationset quietperiod="30">
<svn localWorkingCopy="projects/${project.name}"/>
</modificationset>
<schedule interval="60">
<ant anthome="apache-ant-1.6.5"
buildfile="projects/${project.name}/build.xml"
target="dist-clean" />
</schedule>
<log>
<merge dir="projects/${project.name}/test-results"/>
</log>
<publishers>
<onsuccess>
<artifactspublisher dest="artifacts/${project.name}"
file="projects/${project.name}/dist/lib/AntJUnitExample.jar"/>
</onsuccess>
<htmlemail mailhost="mail.mymailserver.com"
mailport="25"
username="[email protected]"
password="secret"
returnaddress="[email protected]"
defaultsuffix="@mymailserver.com"
logdir="logs/${project.name}"
css="c:\lib\cruisecontrol-bin-2.6\
webapps\cruisecontrol\css\cruisecontrol.css"
xsldir="c:\lib\cruisecontrol-bin-2.6\
webapps\cruisecontrol\xsl"
buildresultsurl="http://localhost:8080/cruisecontrol/
buildresults/${project.name}">
<map alias="steve" address="[email protected]" />
<always address="steve" />
</htmlemail>
</publishers>
</project>
There are several important CruiseControl components in Listing One. The svnbootstrapper bootstrapper plug-in checks for (and checks out if it exists) new code in the Subversion repository. The schedule section controls how often the source repository is checked for new code, and the Ant task to run (30 minutes is actually a more realistic interval for large projects). The publishers section controls what happens after the CruiseControl project has completed its run—in this case copying the newly built JAR file to an artifacts directory if successful, and sending notifications in any case.
With this project defined, you can start CruiseControl by running its cruisecontrol.bat (or cruisecontrol.sh) file. Figure 1 shows its built-in servlet-based web interface, accessed at http://servername:8080.
Now you can add performance tests into the continuous integration process. The most straightforward method is to create a new virtual project in CruiseControl, one that uses the same build script but executes a performance unit-testing target. The following shows the key changes you could make to a copy of Listing One's project definition:
<project name="ant-junit-performance-unit-tests">
...
<schedule interval="600">
<ant anthome="apache-ant-1.6.5"
buildfile="projects/${project.name}/build.xml"
target="performance-unit-tests.execute" />
</schedule>
<publishers>
<onsuccess>
<artifactspublisher dest="artifacts/${project.name}"
dir="projects/ant-junit/profiling-reports"/>
</onsuccess>
...
In this case, CruiseControl is configured to check for source updates every five minutes—because performance tests take longer to run, they cannot and should not be run as frequently. In a real-world application, checking hourly is more reasonable.
But most importantly, a new Ant task, performance-unit-tests.execute, is run, and the results are saved to a profiling-reports directory. The actual definition of this Ant task and the performance profiling process is entirely dependent on the analysis tool used for this. Besides commercial code analysis tools, some promising looking open-source alternatives are emerging, such as EMMA (emma.sourceforge.net).
Similarly, you can perform integration and load testing in this continuous integration environment. The freely available Apache JMeter is my choice for a simple load generator for our web-based application. It's simple enough to integrate JMeter with Ant using an Ant task created by Programmer Planet (programmerplanet.org/media/ant-jmeter/ant-jmeter.jar).
Once integrated, you define a JMeter Ant task by adding the following to the Ant script:
<taskdef name="jmeter" classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>
JMeter test plans are written in XML and they specify the details of the load test. These test plans are called by the JMeter Ant task, which outputs the results to a log file. JMeter test plans can define such metrics as the number of virtual users (threads) and test iterations (loops). You can configure the JMeter Ant task to run different test plans to test basic integration and also to perform full load testing.
Although JMeter outputs a comma-separated values (CSV) file by default, you can change this to XML using the jmeter.save.saveservice.output_format=xml argument. This lets us programmatically determine success or failure, and produce a human-readable report using an XML style sheet (XSL). This report can be evaluated against the application's service-level agreements to give an idea, during development, of whether it meets its scalability requirements. It is possible to further integrate cross-JVM performance analysis tools to help diagnose scalability issues.
Continuous integration is becoming increasingly popular as teams see the benefits of automating more aspects of the development process. Extending continuous integration to different types of performance testing makes the investment in continuous integration even more worthwhile.
Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.