Channels ▼
RSS

Design

Testing for Failures with Python


Unit testing is not just about checking whether a unit under test behaves as designed. It is also about checking how well the unit fails. This article explores how to include failure testing in your Python unit testing. I begin by examining how failures appear in Python, and what Python uses to signal and catch a failure. I will simulate a failure using mocks and patches, mark a test routine for failure, and make a test routine fail on cue. And then I examine how to check for failures, then how to track and collect failure statistics. All sample code requires Python 2.4 (or later) and Mock 1.0.1.

Failures happen, which is why failure testing should be an essential part of any unit test suite. Failure testing helps identify and isolate potential points of failure. Done right, it enables you to intelligently examine each point of failure, add code to catch and process the failure, or rewrite surrounding code to prevent the failure from occurring.

Failures in Python

Python uses exceptions to signal failures. The exception can be predefined or it can be one you define yourself. You can trap the exception with a try…except block or you can hand it over to the next try…except block.

Listing One is a simple try…except block at work. The failure, in this case, is a NameError, caused on line 6 by the rsplit() method trying to process the undefined local fooFail.

Listing One

#!/usr/bin/python

import sys

try:
	fooTest = "Peter Piper picked a peck of pickled peppers."
	fooTest = fooFail.rsplit(" ")
	
	for fooIndex in range(1,5):
		print fooIndex, ":", fooTest[fooIndex]
except IndexError:
	print "failure:index out of range."
	# ...post-failure code follows...
except:
	print "failure:type: ", sys.exc_info()[0]
	print "failure:message: ", sys.exc_info()[1]
	print "failure:traceback: ", sys.exc_info()[2]
	# ...post-failure code follows...
else:
	print "successful run..."
	# ...post-success code follows...
finally:
	print "cleaning up..."
	# ...clean up code follows...

In the example try block, there are two except handlers. The first handler looks for an IndexError signal and tries to make the necessary corrections. The second handler looks for any failure signal other than IndexError. It identifies the failure by checking sys.exc_info. It could try to handle the failure or pass it along by calling raise sys.exc_info[0].

Following the try…except block are two optional blocks. The else block runs only when no failures appear, and the finally block runs regardless of whether a failure occurs.

Mocking Failures

Essential to failure testing is the ability to make the code under test fail on cue. Suppose you are using a mock to represent a test resource. You could induce a failure by setting the mock's side_effect attribute to an incorrect value or an exception. This, in turn, overrides the return_value attribute, which remains set to the correct value or to an error.

Consider the snippet in Listing Two. Class Foo is my test resource, class Bar my test subject. Class Bar has one instance method, doBar(), which creates a Foo instance (barFoo) and invokes the method doFoo() five times (lines 18-23). The doBar() method uses a try…except block (lines 25-33) to catch any failures that crop up. Note the third except handler passes the failure signal to the next try…except block.

Listing Two

import sys
from mock import Mock

# test resource:specification
class Foo(object):
	myFoo = "Foo:myFoo"
	def doFoo(self):
		return("Foo:doFoo:")
	def callFoo(self, anArg):
		print "Foo:callFoo_:", anArg

# test subject
class Bar(object):
	def doBar(self, argFoo):
		# create an instance of Foo
		try:
			
			barFoo = argFoo()
			print barFoo
			
			for barTick in range(5):
				barTest = barFoo.doFoo()
				print barTest
			
		except OverflowError:
			print "failure:buffer:overflow"
		except MemoryError:
			print "failure:out-of-memory"
		except:
			print "failure:", sys.exc_info()[0]
			print "failure:", sys.exc_info()[1]
			print "failure:", sys.exc_info()[2]
			raise (sys.exc_info())
		else:
			print "doBar_:success"
		finally:
			print "doBar_:clean-up"

Listing Three tests how Bar interacts with Foo. I create a mock object (mockFoo), using Foo for a spec (line 3). Then I create a Bar instance (testBar) and pass mockFoo to the instance method doBar() (lines 6-9). Because I did nothing else to mockFoo, testBar behaved as expected.

Listing Three

try:
# prepare the test resource
mockFoo = Mock(spec = Foo)

# prepare the test subject
testBar = Bar()

# run a test
testBar.doBar(mockFoo)
# Output:
# 	<Mock name='mock()' id='428976'>
# 	<Mock name='mock().doFoo()' id='449328'>
# 	<Mock name='mock().doFoo()' id='449328'>
# 	<Mock name='mock().doFoo()' id='449328'>
# 	<Mock name='mock().doFoo()' id='449328'>
# 	<Mock name='mock().doFoo()' id='449328'>
# 	doBar_:success
# 	doBar_:clean-up

except:
	print "failure:", sys.exc_info()

But in Listing Four, I set the side_effect attribute for mockFoo to MemoryError (line 4). When I run the test, the second except handler catches the signal and prints the appropriate message. But what if I change side_effect to EnvironmentError (line 17)? This time, the third except handler catches the signal and prints its type, message, and traceback to stdout. It raises the failure signal, which gets caught by the test's own except handler (line 27-28).

Listing Four

try:
# prepare the test resource
mockFoo = Mock(spec = Foo)
mockFoo.side_effect = MemoryError

# prepare the test subject
testBar = Bar()

# run a test
testBar.doBar(mockFoo)
# Output:
# failure:out-of-memory
# doBar_:clean-up

# change the failure
mockFoo.side_effect = EnvironmentError

# run a test
testBar.doBar(mockFoo)
# Output:
# failure: <type 'exceptions.EnvironmentError'>
# failure: 
# failure: <traceback object at 0x2d5ad0>
# doBar_:clean-up

except:
	print "failure:", sys.exc_info()
# Output:
# failure: (<type 'exceptions.EnvironmentError'>, EnvironmentError(), <traceback object at 0x2d5af8>)

Listing Five shows how I simulate a method failure. When I create the mock object (tempFoo), I set the side_effect attribute for its method doFoo() to OverflowError (lines 3-4). Then I create a second mock object (mockFoo), setting its return_value attribute to tempFoo (line 6). This is to satisfy the constructor call at the beginning of doBar() (see Listing Two, line 18). I create a Bar instance as usual, then pass mockFoo to doBar() (line 12). The result is that doBar() gets an OverflowError signal when it calls doFoo() (see Listing Two, line 22). Its first except handler catches the signal and sends a message to stdout.

Listing Five

try:
# prepare the test resource
tempFoo = Mock(spec = Foo)
tempFoo.doFoo.side_effect = OverflowError

mockFoo = Mock(return_value = tempFoo)

# prepare the test subject
testBar = Bar()

# run a test
testBar.doBar(mockFoo)	
# Output:
# <Mock spec='Foo' id='428016'>
# failure:buffer:overflow
# doBar_:clean-up
	
	# change the failure
	tempFoo.doFoo.side_effect = ["narf", "zort", ReferenceError]
	
	# run a test
	testBar.doBar(mockFoo)
# Output:
# <Mock spec='Foo' id='428016'>
# narf
# zort
# failure: <type 'exceptions.ReferenceError'>
# failure: 
# failure: <traceback object at 0x2d5be8>
# doBar_:clean-up
	
except:
	print "failure:", sys.exc_info()	
# Output:
# failure: (<type 'exceptions.ReferenceError'>, ReferenceError(), <traceback object at 0x2d5cb0>)

What if I set doFoo's side_effect attribute to a list, with the third list item being a ReferenceError (line 19)? This time, doBar() will process the first two responses from doFoo(). Then its third except handler catches the ReferenceError signal. That handler raises the signal, sending it to the test routine's except handler (lines 32-33).

Suppose I have the exception come from the return_value attribute. In Listing Six, I set the return_value attribute for mockFoo to OverflowError (line 3). When I pass mockFoo to doBar(), its third except handler fires to report the failure as an AttributeError failure (lines 16-17).

Listing Six

try:
# prepare the test resource
mockFoo = Mock(spec = Foo, return_value = OverflowError)

# prepare the test subject
testBar = Bar()

# run a test
testBar.doBar(mockFoo)
# Output:
# <type 'exceptions.OverflowError'>
# failure: <type 'exceptions.AttributeError'>
# failure: type object 'exceptions.OverflowError' has no attribute 'doFoo'
# failure: <traceback object at 0x2d5a80>
# doBar_:clean-up
	
except:
	print "failure:", sys.exc_info()
# Output
# failure: (<type 'exceptions.AttributeError'>, AttributeError(), <traceback object at 0x2d5b48>)

In Listing Seven, I set the return_value attribute for the mocked method doFoo(). Now when I pass mockFoo to doBar(), doBar returns five lines of OverflowError. None of doBar's except handlers fired in response.

Listing Seven

try:
	# prepare the test resource
	tempFoo = Mock(spec = Foo)
	tempFoo.doFoo.return_value = OverflowError
	
	mockFoo = Mock(return_value = tempFoo)
	
	# prepare the test subject
	testBar = Bar()
	
	# run a test
	testBar.doBar(mockFoo)
# Output:
# <Mock spec='Foo' id='427952'>
# <type 'exceptions.OverflowError'>
# <type 'exceptions.OverflowError'>
# <type 'exceptions.OverflowError'>
# <type 'exceptions.OverflowError'>
# <type 'exceptions.OverflowError'>
# doBar_:success
# doBar_:clean-up
	
except:
	print "failure:", sys.exc_info()

Both examples treat the exception as an error signal, not a failure one. None of the except handlers fire. If one does, it is because of a different, unrelated failure.

Patching for Failures

You can also use patching to induce a failure. The effect is similar to using a mock object, only the patched mock is temporary and easy to customize. Listing Eight demonstrates how I simulate a failure with a core patch. This test routine is similar to the one in Listing Four. First, I prepared the mock's side_effect attribute as a key/value pair (mockArgs). Then I pass class Foo and mockArgs to the core patch (line 7). Note how I invoked core patch using the with keyword.

Listing Eight

  
try:	
# prepare the test subject
testBar = Bar()

# start patching
mockArgs = {'side_effect':MemoryError}
with patch('__main__.Foo', **mockArgs) as mockFoo:
	# run the test
	testBar.doBar(mockFoo)
# Output
# failure:out-of-memory
# doBar_:clean-up
		
		# change the attribute
		mockFoo.side_effect = EnvironmentError
		
		# repeat the test
		testBar.doBar(mockFoo)
# Output
# failure: <type 'exceptions.EnvironmentError'>
# failure: 
# failure: <traceback object at 0x7a120>
# doBar_:clean-up
	
except:
	print "failure:", sys.exc_info()
# Output
# failure: (<type 'exceptions.EnvironmentError'>, EnvironmentError(), <traceback object at 0x76fa8>)

Core patch returns a mock object (mockFoo), which I then pass to the testBar method doBar() (line 9). Once again, doBar() catches the MemoryError signal with the second except handler.

Next, I changed the side_effect attribute to EnvironmentError (line 15). I passed the updated mockFoo to doBar(). The latter responds by catching the signal with its third except handler, which then sends the signal back to the routine's own except handler (line 25-26).

Listing Nine shows another use of the core decorator. Here, I create the mock object tempFoo and set the side_effect attribute for doFoo to OverflowError (lines 6-7). Then I prepare mockArgs, invoke core patch, and pass class Foo and mockArgs as input (lines 9-10). Inside the with block, I pass the resulting mock (mockFoo) to the doBar() method (line 11).

Listing Nine

try:	
	# prepare the test subject
	testBar = Bar()

	# start patching
	tempFoo = Mock(spec = Foo)
	tempFoo.doFoo.side_effect = OverflowError
	
	mockArgs = {'return_value':tempFoo}
	with patch('__main__.Foo', **mockArgs) as mockFoo:
		testBar.doBar(mockFoo)
# Output
# <Mock spec='Foo' id='430032'>
# failure:buffer:overflow
# doBar_:clean-up
	
	tempFoo.doFoo.side_effect = ["narf", "zort", ReferenceError]
	mockArgs = {'return_value':tempFoo}
	with patch('__main__.Foo', **mockArgs) as mockFoo:
		testBar.doBar(mockFoo)
# Output
# <Mock spec='Foo' id='430032'>
# narf
# zort
# failure: <type 'exceptions.ReferenceError'>
# failure: 
# failure: <traceback object at 0x2d8f80>
# doBar_:clean-up
	
except:
	print "failure:", sys.exc_info()
# Output
# failure: (<type 'exceptions.ReferenceError'>, ReferenceError(), <traceback object at 0x2d8fa8>)

The OverflowError signal trips the first except handler, and doBar() reports the failure just as it did in Listing Five. When I set the same side_effect attribute to a list of values (line 17), both doBar() and the test routine behave as they did in Listing Five.

Listing Ten demonstrates how not to simulate a failure with core patch. What I have here are two nested with blocks. In the outer block (lines 6-7), I set the side_effect attribute for doFoo() to OverflowError. In the inner block (line 8-9), I set the return_value attribute for mockFoo to tempFoo, which is prepared by the outer block. But when I pass mockFoo to the doBar() method, doBar() invokes the doFoo() method five times with nary a problem. It is as if the return_value attribute never had tempFoo as its value.

Listing Ten

try:	
	# prepare the test subject
	testBar = Bar()

	# prepare the test resource
	tempArgs = {'doFoo:side_effect':OverflowError}
	with patch('__main__.Foo', **tempArgs) as tempFoo:
		mockArgs = {'return_value':tempFoo}
		with patch('__main__.Foo', **mockArgs) as mockFoo:
			# run the test
			testBar.doBar(mockFoo)
# Output
# <MagicMock name='Foo' id='449104'>
# <MagicMock name='Foo.doFoo()' id='3061168'>
# <MagicMock name='Foo.doFoo()' id='3061168'>
# <MagicMock name='Foo.doFoo()' id='3061168'>
# <MagicMock name='Foo.doFoo()' id='3061168'>
# <MagicMock name='Foo.doFoo()' id='3061168'>
# doBar_:success
# doBar_:clean-up
	
except:
	print "failure:", sys.exc_info()


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video