Testing Web Applications

Sean and Kristin are senior undergraduate computer science students at the University of Toronto. They can be reached at [email protected] and kristinkerr@ gmail.com, respectively.

Over the past five years, testing has evolved from retroactive (testing at the end of a cycle) to proactive (test-driven development). However, testing web applications is challenging. For Java servlet-based applications, the problem is due in part to the servlet's dependency on the servlet container. Issues arise, such as how testing can be performed inside a container, how servlet requests made from outside the container can be tested, and how a JSP can be tested to ensure that the page is rendered correctly.

Frameworks such as Tapestry (http:// jakarta.apache.org/tapestry/) further complicate testing. While these frameworks simplify the creation of dynamic user interfaces, they make it difficult to isolate areas of code that rely on a large number of components within the framework.

In 2004, we worked on Hippo, a Java web application for managing undergraduate team programming projects. Hippo used the Tapestry web application framework and Hibernate (http://www.hibernate .org/) for managing data persistence. Automated unit testing was a guiding principle throughout development. The creation of new tests came to a standstill, however, when development moved on to the application's web-based interface.

To address this, we set out to extend our test framework so that developers could automate the testing of the presentation layer. In this article, we describe how we integrated JWebUnit (http://jwebunit .sourceforge.net/) into Hippo's existing test framework.

The Testing Toolshed

Servlet applications typically use the Model-View-Controller (MVC) design pattern. The servlet acts as the controller, receiving HTTP requests and dispatching to the appropriate domain objects. The servlet then selects a View (JSP) to render the response, possibly using a templating engine. While there is no one-size-fits-all framework to test these layers, the layered structure lets the testing be broken down into smaller stages.

The Model layer, typically backed by a database, can be isolated and tested using standard unit testing. In this context, a testing framework, such as JUnit, is acting as the Controller; the Model is oblivious to how it is being used.

The Controller layer is harder to test because interacting with the Controller (servlet) requires communicating using HTTP. Servlets usually live in a servlet container that dispatches the HTTP requests to the appropriate servlet. Testing the controller layer directly requires a testing framework, such as Cactus (http://jakarta .apache.org/cactus/), to simulate a servlet container. Testing the View layer can be done in a number of ways. First, you can view the entire application as a black box, sending requests to the servlet container and analyzing the returned response. This requires a testing framework, such as HttpUnit (http://httpunit.sourceforge.net/), to simulate HTTP requests; the generated HTML can then be analyzed by the tester to ensure correctness.

To gain more control, the Model/Controller layers can be set to a predetermined state, at which point the rendered View can be verified. This strategy works well with templating frameworks that use data objects in combination with a set of templates to render the view. The testing framework, rather than the Controller, supplies the data object, allowing for fine-grained testing.

This divide-and-conquer approach to testing is easiest to apply to applications designed with clear separation. For example, object-oriented programming simplifies unit testing by modularizing the code. Likewise, the MVC design pattern takes this idea to a higher level; functionality, rather than code, is modularized. For this reason, web-development frameworks that enforce this separation simplify both development and, in theory, testing.

In practice, however, web-development frameworks can have the opposite effect. Complicated frameworks tend to hide the inner workings of an application, freeing the developer from dealing with mundane and error-prone details. While this is ideal for development, testing requires digging through the abstraction that the framework presents (Law of Leaky Abstractions; http://www.joelonsoftware .com/articles/LeakyAbstractions.html). If the framework is not designed with testing in mind, it may be difficult or impossible to interact with the framework internals. A vicious cycle soon follows, with developers reluctant to test, and frameworks less likely to be designed with testing in mind for lack of demand.

In our case, we found the Tapestry 3.0 framework difficult to test, a point acknowledged by its creator, Howard Lewis Ship. Testing Tapestry code requires working with abstract classes, long lists of mock objects, and monolithic Tapestry code. Howard is planning to improve testing in the upcoming 3.1 release, but this did not help our project at the time.

JWebUnit

Many web-testing tools and frameworks are available, both commercially and through the open-source community. One of the first we looked at was HttpUnit, an open-source Java library that simulates a web browser, allowing external, black-box testing of a web application. By providing an API for interacting with HTTP servers, a live web application can be queried and the response analyzed. This API allows for submitting forms, parsing HTML, following links, and setting cookies. It also offers basic JavaScript support, and separate support for using a simulated servlet container.

Ultimately we chose to use JWebUnit, an open-source tool for automated external testing of web applications. This test framework combines the functionality of HttpUnit with JUnit and includes an extensive set of assertions to analyze the returned HTML. Because JWebUnit extends JUnit, developers familiar with JUnit can start using this web-testing framework right away. The JWebUnit API contains methods to navigate web sites (by links or form submission) and analyze the returned HTML DOM tree.

The two most important classes in JWebUnit are:

• WebTester, which provides a high-level API for basic web-application navigation and validation by wrapping HttpUnit and providing JUnit assertions.
• WebTestCase, which extends JUnit's TestCase class, and acts as a wrapper for a WebTester object.

To implement a test with JWebUnit, you use the WebTester API to navigate to a specific web page, then use the provided assert methods to validate the HTML returned by the server.

Using JWebUnit is much like manual web browsing, except that you can automate the procedure and perform complex validation on the generated HTML. It runs against a live instance of the application to be tested, which means that the application must be deployed to a servlet container during testing. Once JWebUnit is configured to point to the URL of the application, the various navigation and assertion methods can be called to browse the application and verify its operation. For instance, the sequence in Listing One tests the operation of a login form.

Starting at the main page of the application, Listing One looks for fields named username and password, sets values for them, asserts the presence of a button called login, and presses the login button. The form is submitted to the application just as if users had entered and submitted those values from a browser. The returned web page is stored in the WebTester object; consequent actions, such as asserting a link is present, apply to this recently returned page. In this way, writing tests in JWebUnit is fairly intuitive, hiding the complicated details of HTTP requests and responses. Other methods allow assertions to be made about the existence and content of forms, tables, and so on.

Integrating JWebUnit

Prior to writing any tests, we had to modify Hippo's build and test environment. Again, JWebUnit requires a functioning application to test against; the application must be built, initialized, and deployed to a servlet container. The first step was to automate the process of building and deploying to a servlet container so that the test suite could be invoked in a single step. Both Tomcat and Jetty (http:// jakarta.apache.org/cactus/integration/ integration_jetty.html), the servlet containers our team used while developing Hippo, provide Java packages allowing servers to be managed from within an application. This let us create a wrapper around our test suite, automatically starting and stopping the servlet container during the web tests.

Furthermore, building and deploying a web application usually involves combining a number of external components, many of which may be slow, bulky, and time consuming to set up. These issues are less of a concern with standard unit testing, where the functionality being tested tends to be more isolated.

In our case, Hippo requires a functioning Subversion server to provide version control. This places an unwanted dependency on developers wishing to run the web test suite; either a local Subversion server must be running or Hippo must be configured to use a remote Subversion server. The former would be inconvenient and time consuming to set up, the latter slow and dependent on network access.

We removed this dependency by stubbing out the Subversion layer. First, we specified a custom version-control interface; classes implementing this interface are expected to provide only the functionality required by Hippo (for example, creating a repository or adding a file). Interaction between Hippo and the version- control layer occurs through this interface. We then created two version-control implementations: The first encapsulates the functionality provided by the official Subversion Java bindings, while the second is a trivial implementation consisting of empty methods. The version-control implementation is then chosen at runtime using a factory method, letting testing proceed without a functioning Subversion server.

The trivial implementation was sufficient for our purposes because interaction with the Subversion repository was limited to a small section of code. Had it been more widespread, the behavior of Hippo would have changed dramatically during testing. The middleground between the trivial and functional implementations is an implementation that simulates a version-control repository in memory, using a simple tree-based data structure to represent the repository. Version-control functionality can then be tested without having to interact with a slow Subversion server.

It could be argued that using a virtual implementation, whether trivial or simulated, is ineffective—how can we find problems in the application if we remove the functioning components during testing? Well, in some sense we cannot. There are bound to be subtle bugs lurking that only surface when we use the functional component. Such examples include filesystem synchronization, database locking, network failures, and so forth. While a virtual implementation cannot discover these bugs, it can discover those bugs that would surface regardless of the implementation used—the quick and dirty bugs. These bugs are arguably more common, and thus more important to catch in a rapid edit-test-debug development cycle. Because virtual implementations tend to be an order of magnitude faster than their functional equivalent, testing can be done early and often.

Due to the plug-in nature of the implementations, the functional implementation can be transparently substituted into the application, allowing real testing to be carried out. The insidious bugs that may have escaped the initial tests will then be discovered. This approach of "test once, and then test for real" allows testing to be partitioned into stages, particularly important if the testing process is time consuming. We also needed to modify the Tapestry layer of Hippo. JWebUnit uses anchors to identify components in HTML source code. For instance, to locate a table in the source HTML of a web page with the string <table id="table23"...>, the ID table23 is used as an anchor. Tapestry uses static HTML templates as a guide for generating dynamic HTML, but the anchors produced are neither predictable nor consistent.

For example, this static HTML:

<input type="text" jwcid="@TextField" ... />

generates this HTML during runtime by Tapestry:

<input type="text" name="$TextField$0" ... />

The dynamic anchor ($TextField$0) is determined by Tapestry at runtime and changes depending on the composition of the static HTML template. Because Tapestry automatically maps code variables to HTML variables, you aren't required to ensure that anchors in the HTML match those in the code. This makes it difficult to analyze the generated HTML; without consistent anchors, it is impossible to locate tables, forms, and so forth. To get around this, we instructed Tapestry to insert specific anchors. For instance, the revised static HTML:

<input type="text" jwcid="field1@TextField" ... />

generates a consistent anchor in the HTML:

<input type="text" name="field1" ... />

Fortunately, this only required a small amount of changes to the various static templates. Had the previous Hippo team been designing with testing in mind, they would have already added these tags.

The last major change we made was related to cleaning and initializing Hippo's data layer. Effective unit tests are independent; the outcome of any particular test should not be affected by the order in which it is run within the suite. In most cases, independence can be attained by initializing the testing environment to a predetermined state prior to running a test. In fact, this is such a common procedure that JUnit provides built-in methods, setUp() and tearDown(), which are called before and after, respectively, every unit test.

Initializing the environment for an isolated class is usually straightforward. Testing against a live application, however, complicates the matter. We now had to be concerned with reinitializing every component of the data layer (database, subversion repository, and so on) without direct access to any of the internal data structures used in Hippo.

We implemented a set of hooks allowing our external testing framework to trigger the initialization cycle while Hippo was deployed. The easiest way to implement this was by adding two additional submit buttons (reset, rebuild) to the main login form of Hippo that would perform the necessary actions once invoked. JWebUnit could then be configured to click these buttons prior to every web test, allowing each test to run independently of the others.

The obvious drawback to this scheme is the slowdown resulting from a costly initialization cycle every test. This slowdown can be offset somewhat by using mock objects and in-memory data structures whenever possible. For example, some databases can be configured to run entirely in memory rather than constantly reading and writing to disk, a well-known performance bottleneck. This is another example of the "test once, and then test for real" concept we described earlier; in-memory databases may mask subtle bugs, but they are generally much faster than databases on disk.

Implementing the Test Framework

Similar to JUnit, creating test-case classes using JWebUnit requires extending WebTestCase or any class that subclasses it. We chose to extend WebTestCase with a new base class, HippoWebTestCase (see Figure 1). This gave us a place to put common set-up code and utility methods that could be used by all tests.

First, we overrode the setUp() method to handle configuration of the test environment. In particular, JWebUnit was configured to point to the application's base URL. Since we deployed the application on our own local machines for testing purposes, we set the base URL to "http:// localhost:8080." We also used the setUp() method to handle cleaning and initializing of the database, as discussed in the previous section.

Then we added a number of utility methods to handle common testing scenarios. For example, we created a login method that would execute the sequence of steps required to login to Hippo (go to the login page, fill in form parameters, and submit the form). Other methods were created to handle common navigation sequences, such as navigating to the main Project configuration page, a prerequisite for doing any project-related testing.

We decided to start all navigation sequences at the login page, regardless of the current position within the web page. Because the front end of the application is built on Tapestry, which generates HTML pages dynamically at runtime, we could not always predict the current state of the application. By beginning at the login page, we could help ensure consistent navigation sequences. Also, jumping directly to a specific web page was not possible as Tapestry performs a complex URL transformation depending on dynamic content. Unfortunately, this adds to the running time of the tests, as each navigation sequence requires that much more data to be sent over the network. Listing Two is an example of a typical navigation sequence.

This method reproduces the sequence of steps a user would take when manually moving through Hippo. The followLink(String linkId) method is a custom Hippo method that uses the JWebUnit API (http://jwebunit.sourceforge.net/apidocs/ index.html) to first assert that the link is present with a text value of linkId, then calls clickLinkWithTest(linkId). Finally, we added a number of operations that could be viewed as atomic to the developer. For example, you may wish to add a project and then assert that the project was indeed added. Listing Three executes the sequence of actions to add a project to Hippo.

Now, a test class can be created by extending HippoWebTestCase and creating test methods using these utility methods and the JWebUnit API to browse Hippo, execute actions, and make assertions about the results.

Conclusion

By using JWebUnit, we can now perform functional testing on Hippo. We have only addressed one facet of testing, so our work isn't quite done yet. Our decision to use JWebUnit was primarily influenced by the difficulty associated with testing a Tapestry application. While Tapestry is a terrific framework to develop with, the testing support needs to be improved.

Our testing framework is not perfect. If you're accustomed to running hundreds of unit tests in a few seconds, you may be frustrated when running JWebUnit tests. As discussed, JWebUnit requires interacting with an application over a network. Even on a local network, the time required to send requests between the testing framework and the application quickly accumulates. On top of this, the application itself may have to perform time-consuming tasks, such as interacting with the filesystem or querying a subversion repository. For example, each JWebUnit test in our framework takes, on average, one to two seconds to complete.

Given hundreds of tests, it's impractical to frequently run the entire test suite. Moreover, JWebUnit presents some tradeoffs developers will have to live with. Using a functional testing framework like JWebUnit is arguably less beneficial to the developer than standard unit testing with JUnit.

This is because higher level approaches to testing make it difficult to test functionality in isolation; as a result, it's not always self-evident what went wrong in a JWebUnit test because the entire application is working in concert. On the other hand, functional testing frameworks allow the developer to simulate user interaction with the application. A comprehensive suite of JWebUnit tests can, therefore, save you countless hours of manual testing.

The fact that our framework has its problems points to the central issue—there is no single solution for testing. Rather, testing often requires many different strategies to fully exercise an application. Testing a web application presents more challenges, but applications designed with testing in mind will go a long way to encouraging testing in the future.

DDJ

Listing One

getTestContext().setBaseUrl("http://localhost");
beginAt("/main");
setField("username", username);
setField("password", password);
assertSubmitButtonPresent("login");
assertSubmitButtonPresent("reset");
submit("login");

private void gotoProjectAdmin() {
   login();
   followLink("System Administration");
   followLink("Manage Projects");
}

protected void addProject(String parentProject, String project, String url) {
   gotoProjectAdmin();
   followLink("Create a project");
   assertFormPresent();
   setField("projectName", project);
   setField("parentProject", parentProject);
   setField("courseURL", url);
   submit("create");
}