Channels ▼
RSS

Design

Testing for Failures with Python


In Listing Nineteen, however, I gave ReferenceError a different failure message (line 8). Now assertRaisesRegexp fires, because the text pattern "\Aincorrect" does not match the failure message (line 14).

Listing Nineteen

class BarTest(unittest.TestCase):
    
    def test_bar(self):
        print "Running the test..."
        
        # prepare the test resource
        mockFoo = Mock(spec = Foo)
        mockFoo.side_effect = ReferenceError("incorrect object reference")
        
        # prepare the test subject
        testBar = Bar()
        
        # run a test
        with self.assertRaisesRegexp(ReferenceError, "\Amissing") as fooErr:
            testBar.doBar(mockFoo)
# Result:
# failure: <type 'exceptions.ReferenceError'>
# failure: incorrect object reference
# failure: <traceback object at 0x46c940>
# doBar_:clean-up

# run the test
if __name__ == '__main__':
    unittest.main()
# Result:
# AssertionError: "\Amissing" does not match "incorrect object reference"
# F
# FAIL: test_bar (__main__.BarTest)
# FAILED (failures=1)

In all four examples, I paired the asserts with a with keyword and placed the calls to the test subject within the with block. If a method call fails, any failure signal it raises will be seen by the assert.

The same two asserts have a variant, one geared towards method failures. This variant takes three more arguments: the method name, an arguments list, and a key/value list. The last two arguments are optional if the target method does not accept input.

assertRaises(failure-signal, method-name, 
	*arguments-list, **key/value-list)
assertRaisesRegexp(failure-signal, text-pattern, method-name, 
	*arguments-list, **key/value-list)

Listing Twenty demonstrates how assertRaises checks for method failure. Again, I set the side_effect attribute for mockFoo to ReferenceError (line 8). I create an arguments list (testArgs), with mockFoo as the sole entry (line 14). When I invoke assertRaises, I pass ReferenceError as the failure signal, testBar.doBar the test method, and testArgs (line 15). Note the exclusion of () and the use of dot-notation to invoke the method.

Listing Twenty

class BarTest(unittest.TestCase):
    
    def test_bar(self):
        print "Running the test..."
        
        # prepare the test resource
        mockFoo = Mock(spec = Foo)
        mockFoo.side_effect = ReferenceError
        
        # prepare the test subject
        testBar = Bar()
        
        # run a test
        testArgs = [mockFoo]
        self.assertRaises(ReferenceError, testBar.doBar, *testArgs)
# Result:
# failure: <type 'exceptions.ReferenceError'>
# failure: 
# failure: <traceback object at 0x46b940>
# doBar_:clean-up

# run the test
if __name__ == '__main__':
    unittest.main()

As usual, doBar() catches the ReferenceError signal with the third except handler. It passes on the signal, assertRaises catches it and does not fire.

In Listing Twenty-one, I set the same side_effect attribute to MemoryError (line 8). Now, doBar() catches the signal with the second except handler. However, the handler does not pass on MemoryError. So assertRaises fires (line 15) to report the "handled" failure.

Listing Twenty-one

class BarTest(unittest.TestCase):
    
    def test_bar(self):
        print "Running the test..."
        
        # prepare the test resource
        mockFoo = Mock(spec = Foo)
        mockFoo.side_effect = MemoryError
        
        # prepare the test subject
        testBar = Bar()
        
        # run a test
        testArgs = [mockFoo]
        self.assertRaises(MemoryError, testBar.doBar, *testArgs)
# Result:
# failure:out-of-memory
# doBar_:clean-up

# run the test
if __name__ == '__main__':
    unittest.main()
# Result:
# F
# FAIL: test_bar (__main__.BarTest)
# FAILED (failures=1)

Reporting on Failures

Finally, failure results are conveyed by three TestResult properties. The property failures reveal the failed test routines. The property expectedFailures reveal those routines marked for failure. And the property unexpectedSuccesses reveal those routines that are supposed to fail but did not. All three properties return a list of tuples. The first half of each tuple names the TestCase instance and the test routine. The second half is a traceback showing the test file and the point of failure.

Listing Twenty-two demonstrates how to access one of the properties. My test routines are the same test_barA and test_barB from Listing Fifteen. I use the TestLoader class to turn BarTest into the test suite barSuite (line 31). Then I create an instance of TextTestRunner (line 8) and pass barSuite into the run() method (line 10). The run method returns a TestResults object (barResult), and I read the failures property (lines 10, 15). I then print out the tuple values from the failures property to stdout (lines 20-22).

Listing Twenty-two

class BarTest(unittest.TestCase):

# see Listing Fifteen...
    
# run the test
if __name__ == '__main__':
    barSuite = unittest2.TestLoader().loadTestsFromTestCase(BarTest)
    barRunner = unittest2.TextTestRunner(descriptions=True)
    
    barResult = barRunner.run(barSuite)
# Result:
# Fx
# FAIL: test_barA (__main__.BarTest)
# FAILED (failures=1, expected failures=1)
	
    print barResult
# Result:
# <unittest2.runner.TextTestResult run=2 errors=0 failures=1>
    
    barFail = barResult.failures
    print barFail
# Result:
# [(<__main__.BarTest testMethod=test_barA>, 'Traceback (most recent call last):\n  File "/Projects/BarTest.py", line 70, in test_barA\n    self.assertGreater(len(locText), 1, "Incorrect word count")\nAssertionError: 1 not greater than 1 : Incorrect word count\n')]
    
    failItem = barFail[0]
    print failItem[0]
    print failItem[1]
# Result:
# test_barA (__main__.BarTest)
# Traceback (most recent call last):
#   File "/Projects/BarTest.py", line 70, in test_barA
#     self.assertGreater(len(locText), 1, "Incorrect word count")

It is worth mentioning the failfast property. This one controls how the whole test case reacts to a failure. Set failfast to False (default), and the test case runs all its test routines even when one of them failed. Setting failfast to True, and the test case stops after the first failed test routine.

Listing Twenty-three demonstrates the failfast property at work. Again, my test routines are the same ones from Listing Fifteen. But when I invoke unittest.main() (line 31), I pass a failfast argument with a True value. BarTest then reacts by stopping right after test_barA fails, leaving test_barB untouched. Should I pass failfast with a False argument, BarTest will run test_barB even after test_barA has failed.

Listing Twenty-three

class BarTest(unittest.TestCase):
    
    # see Listing Fifteen...
    
# run the test
if __name__ == '__main__':
    unittest.main(failfast = True)
# Results
# F
# FAIL: test_barA (__main__.BarTest)
# FAILED (failures=1)

Failures in Unit Tests

Let's move on to an example of how to include failure testing in a typical unit test setup. Our test setup is the same one described by Martin Fowler in his seminal article Mocks Aren't Stubs. The Order class is the test subject, the Warehouse class is the test resource, and the OrderTest class is the test case.

To see the entire updated unit test, download the script (OrderTestFail.py).

In Listing Twenty-four is the original test routine testA1_newOrder. Point of this test is to check if the Order constructor returns a valid object. Both the except block (line 13-14) and the three asserts (lines 15-25) should not fire during a test run.

Listing Twenty-four

  class OrderTest(unittest.TestCase):
    
    # see demo test script...
        
    # test: new order
    # objective: creating an order
    def testA1_newOrder(self):
        """Normal test"""
        # creating a new order
        try:
            testOrder = Order("mushrooms", 10)
            print str(testOrder)
        except:
            print "failure:", sys.exc_info()[0]
        finally:                    
            # test for a nil object
            self.assertIsNotNone(testOrder, "Invalid order")
            
            # test for a valid item name
            testName = testOrder._orderItem
            self.assertEqual(testName, "mushrooms", "Invalid item name")
            
            # test for a valid item amount
            testAmount = testOrder._orderAmount
            self.assertGreater(testAmount, 0, "Invalid item amount")
    
    # continued in Listing Twenty-five...

Listing Twenty-five shows a variation of the above routine (testA2_newOrder). In this one, I pass incorrect arguments to the Order constructor (line 10). The constructor returns a None, and the except handler fires. Also, the asserts in the finally block do not fire (lines 17-25), because testOrder meets the expected conditions.

Listing Twenty-five

class OrderTest(unittest.TestCase):
    
# see Listing Twenty-four...    

@unittest.expectedFailure
def testA2_newOrder(self):
	"""Modified for failure"""
	# creating a new order
	try:
		testOrder = Order("mushrooms", "ten")
		print str(testOrder)
	except:
		print "failure:", sys.exc_info()[0]
		print "failure:", sys.exc_info()[1]
	finally:
		# test for a nil object
		self.assertIsNotNone(testOrder, "Invalid order")
		
		# test for a valid item name
		testName = testOrder._orderItem
		self.assertEqual(testName, "mushrooms", "Invalid item name")
		
		# test for a valid item amount
		testAmount = testOrder._orderAmount
		self.assertGreater(testAmount, 0, "Invalid item amount")

# continued in Listing Twenty-six...

Listing Twenty-six is the test routine testB1_nilInventory. Point of this test is to see how an Order object (testOrder) reacts to a nil inventory. The Warehouse mock (testMock) returns a None for its constructor call, and testOrder responds by not fulfilling the order.

Listing Twenty-six

class OrderTest(unittest.TestCase):

# see Listing Twenty-five...    

# test: nil inventory
# objective: how the order object handles a nil inventory
def testB1_nilInventory(self):
	"""Normal test"""
	# create the test resource
	testMock = Mock(spec = Warehouse, return_value = None)
	
	# create a new order
	testOrder = Order("mushrooms", 10)
	print str(testOrder)
	
	# fill the order
	testSource = testMock()
	testOrder.fill(testSource)
	
	# perform the checks
	self.assertEqual(testOrder._orderFilled, -1, "The requested amount was obtained")

# continued in Listing Twenty-seven...

Listing Twenty-seven shows the routine modified for failure (testB2_badInventory). Here, I set the side_effect attribute of testMock to MemoryError (line 8), and I placed the creation of testOrder and the call to fill() within a with block (lines 10-18). The result: assertRaises catches the MemoryError signal and assertEqual does not fire.

Listing Twenty-seven

class OrderTest(unittest.TestCase):

# see Listing Twenty-six...    

def testB2_badInventory(self):
	"""Modified for failure"""
	# create the test resource
	testMock = Mock(spec = Warehouse, return_value = None, side_effect = MemoryError)
	
	with self.assertRaises(MemoryError):
		# create a new order
		testOrder = Order("mushrooms", 10)
		print str(testOrder)
		
		# fill the order
		testSource = testMock()
		print testSource
		testOrder.fill(testSource)
		
	# print the mocked calls
	self.assertEqual(testOrder._orderFilled, -1, "The requested amount was obtained")

# continued in Listing Twenty-eight...


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.