Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼


Project of the Month: TestNG

JUnit is a great framework for unit testing that was never designed for the broader kind of testing that I describe in this article. As its name implies, JUnit focuses on testing classes at the unit level, without any dependencies on external resources. By design, JUnit's scope is minimalistic and it offers the smallest set of features necessary to achieve this goal.

Before TestNG appeared in 2004, a lot of Java developers started using JUnit to do much more than it was designed for, and they realized that they needed to extend or tweak JUnit in order to achieve these goals. A simple example of this was the inability to write configuration code for your classes (what we call today "@BeforeClass" and "@AfterClass").

Here is another often misunderstood aspect of JUnit. Consider the following JUnit code:

public class MyTest {
  private int n = 0;

  public void test1() {
    assert n == 1;

  public void test2() {
    assert n == 1;

You might be surprised to know that this test passes because JUnit reinstantiates your test class before each test method in order to make sure that your tests do not depend on any state — so that they are truly independent. Because of this, the field n is reinitialized to zero before each test method starts.

This design decision makes sense for unit tests, but it quickly gets in your way when you start writing functional tests that create an expensive state that needs to be maintained from one test to the next. The typical workaround for this is to make your fields static, but at that point, you're beginning to fight the framework, instead of being helped by it.

TestNG addresses many of these concerns and provides numerous features to assist developers writing functional tests. I will cover some of these features in this article.

A Brand New World

Before we start, I'd like to make a small clarification: In this article, I use the adjective "functional" to mean pretty much anything that's not unit testing. This encompasses all kinds of testing: integration, end-to-end, system, and so on.

Unit and functional testing inhabit two very different worlds &mdash not just in what they test, but also in how the tests are implemented. Here are some of the guidelines that you should usually keep in mind when writing unit tests.

Unit tests should:

  • Run "fast,"
  • Test a small piece of code (typically, a class),
  • Not depend on external resources (for example, a database, the network, and so on.),
  • Be runnable in any order.

I am being intentionally vague in these guidelines because they are pretty loose and I don't think any of them should be taken completely literally. You shouldn't hesitate to break any of these rules if you think the end result will improve the testing of your code base (for example, some of my unit tests actually test more than one class, others depend on an in-memory database).

Functional tests, on the other hand, should…well, there are no rules, really. Everything is fair in functional testing. If you are providing a certain functionality to users in your product, you should do whatever you can to have an automated test of this feature. The resulting test might be "slow" (a very relative measure, obviously, and while I have seen functional tests take tens of minutes to run, I wouldn't expect most functional tests to run longer than a minute). Functional tests might depend on a lot of external resources and test a vast array of your code.

Unlike unit tests, which mock a lot of the code paths they go through, functional tests are running the exact same code that you will be presenting to your users; hence, the increased difficulty in getting them right.

Let's now turn our attention to a few specific features of TestNG that will assist you in this goal, namely:

  • groups
  • data providers
  • parallelism
  • dependencies.


The source code for functional tests can become very, very large. And if you take testing seriously, this code base can even exceed the code base of your project itself — not just because of the scope of what you are testing, but also because setting up your tests so that they are running on an environment that is similar to the one you run in production can result in a lot of supporting infrastructure.

Depending how complex the project is, you will typically end up with tests that cover all kinds of different areas: persistence (relational, NoSQL, other), security, login, front end (UI or native), networking, clustering, failover, caching, and so on. Let's say that you just added functionality in the persistence layer and you'd like to run a quick sanity check to make sure that you didn't break anything vital before moving forward. The unit tests run fast and quickly show you a 100% success rate, but running the entire functional test stack might take hours. Ideally, you'd like to run only the persistence-related functional tests.

This is where TestNG's groups come in handy.

With TestNG, every test method can belong to one or several groups. Here is an example:

@Test(groups = { "functional", "persistence", "slow" })
public void saveShouldWork() { ... }

@Test(groups = { "functional", "persistence", "fast" })
public void databaseShouldBeRunning() { ... }

Once I have compiled my tests, I can ask TestNG to run all the "persistence" tests (which means both methods will be run) or only the "fast" tests (in which case, only databaseShouldBeRunning()runs). This is one of the main benefits of Java's annotations: They are persisted with the class files, which means that TestNG can easily separate the "static" model of your tests (the code) from the "dynamic" aspect (determining which tests to run). You compile once, and then you can run various combinations as often as you need without having to recompile anything.

You can also ask TestNG to exclude groups, which makes it easy to put tests that you know are not going to pass in a specific group (e.g. "broken"), and then exclude that group from the test. When the time comes to release, you make sure that the "broken" group is empty (TestNG will tell you that), and you will never accidentally ship software with vital tests disabled.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.