Channels ▼
RSS

Open Source

Unit Testing with Python


To skip a test routine conditionally, use the class methods unittest.skipIf() and unittest.skipUnless() as in Listing Twelve:

Listing Twelve.

@unittest.skipIf(self.fooC > 456, "Skip over this routine")
def testB():
	"""Test routine B"""
	# the following code will run only when fooC is less
	# than or equal to 456
	fooA = narf()
	self.assertNotEqual(fooA, 123)

@unittest.skipUnless(self.fooC > 456, "Skip over this routine")
def testC():
	"""Test routine C"""
	# the following code will run only when fooC is
	# greater than 456
	fooA = zort()
	self.assertNotEqual(fooA, 123) 

The first method causes a skip when a condition is met, the second when a condition is not met. Both get two arguments: the condition and the reason for the skip. Both must be placed before the test routine. The condition may involve a class property or another class method.

To fail a test routine, use the instance method fail() as in Listing Thirteen:

Listing Thirteen.

def testB():
	"""Test routine B"""
	self.fail("Force this routine to fail.")
	
	# the following code will not run
	fooA = narf()
	self.assertNotEqual(fooA, 123)

def testC():
	"""Test routine C"""
	print "This routine still runs after testB."

This method takes a log message explaining the reason for failure. Like skipTest(), the fail() method goes inside the test routine. Code placed after the call to fail() will not run, but test routines after the failed routine still get to run.

Viewing the Test Results

There are two possible forms of output from the TextTestRunner: console text or a TestResult object. First, let's look at the console output, which shows each test's result. You can control this output by passing three optional arguments to the class constructor.

unittest.TextTestRunner(stream=sys.stderr, descriptions=True, verbosity=1)

The first argument (labelled stream) sets the output destination. This defaults to sys.stderr if one is not specified. Next argument (labelled descriptions) controls how errors and failures are reported. Passing a True (default) tells the runner to name those routines that erred, failed, or skipped. Passing a False tells it not to.

The last constructor argument (labelled verbosity) sets the level of detail. There are three possible levels. For a verbosity of 0, the results show only the number of executed tests and the final outcome of those tests. For a verbosity of 1, the results marks a successful test with a dot, a failed one with an F, a skipped one with an s, and an erroneous one with an E. And for a verbosity of 2, the results lists each test case and test routine, plus the outcome of each routine.

To demonstrate, let's run the test script in Listing Fourteen:

Listing Fourteen: Test script.

import sys
import unittest

class FooTest(unittest2.TestCase):
    """Sample test case"""
    
    # preparing to test
    def setUp(self):
        """ Setting up for the test """
        print "FooTest:setUp_"
    
    # ending the test
    def tearDown(self):
        """Cleaning up after the test"""
        print "FooTest:tearDown_"
    
    # test routine A
    #@unittest2.skip("FooTest:test_A:skipped")
    def test_A(self):
        """Test routine A"""
        self.skipTest("FooTest:test_A:skipped")
        print "FooTest:test_A"
    
    # test routine B
    def test_B(self):
        """Test routine B"""
        fooA = 123
        fooB = 234
        self.assertEqual(fooA, fooB, "A is not equal to B")
        print "FooTest:test_B"
    
    # test routine C
    def test_C(self):
        """Test routine C"""
        fooA = 123
        self.assertEqual(fooA, fooB, "A is not equal to B")
        print "FooTest:test_C"
    
    # test routine D
    def test_D(self):
        """Test routine D"""
        self.fail("FooTest:test_D:fail_")
        print "FooTest:test_D"

# Run the test case
if __name__ == '__main__':
    fooSuite = unittest.TestLoader().loadTestsFromTestCase(FooTest)

If you create the test runner as follows:

fooRunner = unittest.TextTestRunner(description=True)
fooRunner.run(fooSuite)

The results appear as shown next:

sFEF
================================================================
ERROR: test_C (__main__.FooTest)
Test routine C
----------------------------------------------------------------

================================================================
FAIL: test_B (__main__.FooTest)
Test routine B
----------------------------------------------------------------

================================================================
FAIL: test_D (__main__.FooTest)
Test routine D
----------------------------------------------------------------

----------------------------------------------------------------
Ran 4 tests in 0.004s

FAILED (failures=2, errors=1, skipped=1) 

But if you create the runner as follows:

fooRunner = unittest.TextTestRunner(description=False)

The docstr comment line from each test routine will not be included.

The console output above has a verbosity level of 1. It starts with the line sFEF, which states two failed tests, one skipped, and one erroneous test. But suppose you created the test runner by typing

fooRunner = unittest.TextTestRunner(verbosity=0)

Then, the sFEF line will not appear. If you create the test runner by typing

fooRunner = unittest.TextTestRunner(verbosity=2)

Then the console output appears as:

test_A (__main__.FooTest)
Test routine A ... skipped 'FooTest:test_A:skipped'
test_B (__main__.FooTest)
Test routine B ... FAIL
test_C (__main__.FooTest)
Test routine C ... ERROR
test_D (__main__.FooTest)
Test routine D ... FAIL

================================================================
ERROR: test_C (__main__.FooTest)
Test routine C
----------------------------------------------------------------

================================================================
FAIL: test_B (__main__.FooTest)
Test routine B
----------------------------------------------------------------

================================================================
FAIL: test_D (__main__.FooTest)
Test routine D
----------------------------------------------------------------

----------------------------------------------------------------
Ran 4 tests in 0.009s

FAILED (failures=2, errors=1, skipped=1)

Note that the sFEF line is replaced by four pairs of lines, each pair naming the test case and routine, the final test state, the docstr comment of each routine, and any assert messages.

The second type of output is the TestResults object. This one is returned by the runner object after all the tests have run:

fooResult = fooRunner.run(fooSuite)

The TestResult object has two sets of property accessors (Figure 6).

Python Unit Tests
Figure 6: The TestResult object (in part).

This is not a complete set of accessors, but these are ones you will most likely use. The first set returns a list of tuples. Each tuple reveals how each test routine fared when executed. The errors accessor lists identified routines that raised an exception. Each tuple has the name of the test case and routine, the location of the test script, the line position of the error, a trace-back, and a reason for the error.

The failures accessor lists test routines that failed. Its tuples contain the same information as tuples from the errors accessor. The skipped accessor lists routines that were skipped, conditionally or not. Its tuples name the test case and routine, and give the reason for the skip.

The second set of accessors provides additional data. testsRun gives the number of test routines that ran, regardless of outcome. And wasSuccessful() returns a True if all routines ran without problems, False if at least one routine had a problem. Notice this last accessor is written as a function.

Listing Fifteen demonstrates how the TestResult object works.

Listing Fifteen.

import sys
import unittest

class FooTest(unittest.TestCase):
    """Sample test case"""
    
# Run the test case
if __name__ == '__main__':
    fooSuite = unittest.TestLoader().loadTestsFromTestCase(FooTest)
    
    fooRunner = unittest.TextTestRunner()
    fooResult = fooRunner.run(fooSuite)
    
    print 
    print "---- START OF TEST RESULTS"
    print fooResult
    print
    print "fooResult::errors"
    print fooResult.errors
    print
    print "fooResult::failures"
    print fooResult.failures
    print
    print "fooResult::skipped"
    print fooResult.skipped
    print
    print "fooResult::successful"
    print fooResult.wasSuccessful()
    print
    print "fooResult::test-run"
    print fooResult.testsRun
    print "---- END OF TEST RESULTS"
    print

This script uses the same FooTest case defined in Listing Fourteen. After it invokes the run() method in the runner object fooRunner, the script stores the results into the local fooResults. Then it invokes each accessor and prints the test result on the console window.

Here are the test results returned by fooRunner:

---- START: Test Results:
<unittest2.runner.TextTestResult run=4 errors=1 failures=2>

fooResult::errors
[(<__main__.FooTest testMethod=test_C>, 'Traceback (most
recent call last):\n  File "/Volumes/Projects/Pro_Articles/_ddj/
18_pyUnitTest/ddj18_code/foo_testRun.py", line 49, in test_C\n    
self.assertEqual(fooA, fooB, "A is not equal to B")\nNameError: 
global name \'fooB\' is not defined\n')]

fooResult::failures
[(<__main__.FooTest testMethod=test_B>, 'Traceback (most 
recent call last):\n  File "/Volumes/Projects/Pro_Articles/_ddj/
18_pyUnitTest/ddj18_code/foo_testRun.py", line 41, in test_B\n    
self.assertEqual(fooA, fooB, "A is not equal to B")
\nAssertionError: 123 != 234 : A is not equal to B\n'), 
(<__main__.FooTest testMethod=test_D>, 'Traceback (most 
recent call last):\n  File "/Volumes/Projects/Pro_Articles/_ddj/
18_pyUnitTest/ddj18_code/foo_testRun.py", line 55, in test_D\n    
self.fail("FooTest:test_D:fail_")\nAssertionError: 
FooTest:test_D:fail_\n')]

fooResult::skipped
[(<__main__.FooTest testMethod=test_A>, 'FooTest:test_A:skipped')]

fooResult::successful?
False

fooResult::test-run
4

The second line summarizes the results. It shows the total number of test routines and how many routines have erred or failed. The paragraph below it reveals the erroneous routine: test_C(). It also reveals the cause of the error: the undefined variable fooB.

The next clump of text reveals the failed routines: test_B() and test_D(). It reveals why test_B() failed: two unequal values passed to assertEqual(). And it reveals that test_D() has explicitly called the instance method fail().

Below that is shown the skipped routine, test_A(). The next line shows what problems were encountered. And the last two lines report that a total of four test routines ran, confirming what was reported initially.

Conclusion

Python has substantial resources to enable unit testing. In this article, I looked into the unittest module and examined those classes essential for unit testing. I showed how to create a test case, how to design the test routines, and how to gather several test cases into one test suite.

The unittest module can do more than just run basic tests. With it, we can design tests that focus on exceptions or tests that do pattern matching. We can load and execute tests stored in separate files. We can even design tests that use simulated objects (known as mocks). But these are topics for a later article. Meanwhile, the "Recommended References" can provide additional guidance on these more-advanced features.

Recommended References

Test-Driven Development in Python, Jason Diamond. ONLamp.com/O'Reilly Publishing.

Introduction to unittest, Michael Foord. Voidspace.

unittest: Automated testing framework, Doug Hellman. PyMOTW.

unittest: Unit testing framework, Python 2.7.6 documentation.

Testing Your Code, Kenneth Reitz. The Hitchhiker's Guide to Python.


José R. C. Cruz is a freelance technical writer based in British Columbia. He frequently contributes articles to Dr. Dobb's, MacTech, and REALStudio Developer.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video