Channels ▼
RSS

Design

Testing for Failures with Python


Listing Twenty-eight is the test routine testC1_orderCheck. This one checks if the Order object queries the Warehouse mock before filling the item order. It has two subtests, both using patch decorators to prepare the mock. The first subtest simulates a successful query, the second an unsuccessful one.

Listing Twenty-eight

class OrderTest(unittest.TestCase):

# see Listing Twenty-seven...

# test: checking the inventory
# objective: does the order object check for inventory?
def testC1_orderCheck(self):
	"""Normal test"""
	# perform the first test
	testOrder = Order("mushrooms", 10)
	print str(testOrder)
	
	mockArgs = {'hasInventory.return_value':True, 'getInventory.return_value':0}
	with patch('__main__.Warehouse', **mockArgs) as testMock:
		testOrder.fill(testMock)
		
		# perform the checks 
		self.assertFalse(testOrder.isFilled(), "The order got filled")
		self.assertEqual(testOrder._orderFilled, 0, "Incorrect amount obtained")
		
		testMock.hasInventory.assert_called_once_with("mushrooms")
	
	# perform the second test
	testOrder = Order("cabbage", 10)
	print str(testOrder)
	
	with patch('__main__.Warehouse') as testMock:
		# configure the test resource
		testMock.hasInventory.return_value = False
		testMock.getInventory.return_value = 0
		
		# perform the test
		testOrder.fill(testMock)
		
		# perform the checks            
		self.assertFalse(testOrder.isFilled(), "The order got filled")
		self.assertEqual(testOrder._orderFilled, -1, "The requested amount was obtained")
		
		testMock.hasInventory.assert_called_once_with("cabbage")

# continued in Listing Twenty-nine...

Listing Twenty-nine shows the above routine modified for failure (testC2_orderCheck). Here, I cut the number of subtests to 1, and I set the side_effect attribute for the hasInventory() method to LookupError (line 11). I enclosed the test code in a with block, using assertRaises to catch the failure signal (line 13). When I run this test, assertRaises fires, but the later three asserts (lines 17-20) do not.

Listing Twenty-nine

class OrderTest(unittest.TestCase):

# see Listing Twenty-eight...

def testC2_orderCheck(self):
	"""Modified for failure"""
	# perform the test
	testOrder = Order("mushrooms", 10)
	print str(testOrder)
	
	mockArgs = {'hasInventory.return_value':True, 'hasInventory.side_effect':LookupError, 'getInventory.return_value':0}
	with patch('__main__.Warehouse', **mockArgs) as testMock:
		with self.assertRaises(LookupError):
			testOrder.fill(testMock)
			
			# perform the checks 
			self.assertFalse(testOrder.isFilled(), "The order got filled")
			self.assertEqual(testOrder._orderFilled, 0, "The requested amount was obtained")
			
			testMock.hasInventory.assert_called_once_with("mushrooms")

# continued in Listing Thirty...

Finally, Listing Thirty shows the original test routine testD1_orderFilled. The point of the test is to check if the Order object fulfills an item order. The Warehouse mock (testMock) returns a True with its hasInventory() method and a 10 with getInventory() (lines 15-16). When testOrder interacts with testMock, the first two asserts pass (lines 23-24) because testOrder completes the item order correctly. The second two asserts also pass (lines 26-27) because testOrder invoked the right mocked methods with the right arguments.

Listing Thirty

class OrderTest(unittest.TestCase):
    
# see Listing Twenty-nine...

# test: filling the order
# objective: is the order completely filled?
@patch('__main__.Warehouse')
def testD1_orderFilled(self, testMock):
	"""Normal test"""
	# creating a test order
	testOrder = Order("mushrooms", 10)
	print str(testOrder)
	
	# configure the mock
	testMock.hasInventory.return_value = True
	testMock.getInventory.return_value = 10
	
	# perform the test
	testOrder.fill(testMock)
	print testOrder.isFilled()
	
	# perform the checks
	self.assertTrue(testOrder.isFilled(), "The order was not filled")
	self.assertEqual(testOrder._orderFilled, 10, "The requested amount was not obtained")
	
	testMock.hasInventory.assert_called_once_with("mushrooms")
	testMock.getInventory.assert_called_with("mushrooms", 10)

# continued in Listing Thirty-one...
    

Listing Thirty-one shows the above test, now modified for failure (testD2_orderFilled). In this one, I set the return_value attribute for getInventory() to -10. When I pass testMock to the fill() method, the method updates the property _orderFilled, causing assertGreater to fire (line 21). The routine gets an 'F' tag, and it skips the last two asserts (lines 23-24).

Listing Thirty-one

class OrderTest(unittest.TestCase):

# see Listing Thirty...

@patch('__main__.Warehouse')
def testD2_orderFilled(self, testMock):
	"""Modified for failure"""
	# creating a test order
	testOrder = Order("mushrooms", 10)
	
	# configure the mock
	testMock.hasInventory.return_value = True
	testMock.getInventory.return_value = -10
	
	# perform the test
	testOrder.fill(testMock)
	print testOrder.isFilled()
	
	# perform the checks
	self.assertTrue(testOrder.isFilled(), "The order was not filled")
	self.assertGreater(testOrder._orderFilled, -1, "The requested amount was not obtained")
	
	testMock.hasInventory.assert_called_once_with("mushrooms")
	testMock.getInventory.assert_called_with("mushrooms", -10)

# continued in Listing Thirty-two...

Listing Thirty-two shows another variation of the same test (testD3_orderFilled). This time, I mark the routine with the decorator expectedFailure (line 6). I also set the side_effect attribute for getInventory() to LookupError (line 16). When I pass testMock to the fill() method, the method's except handler catches the failure and sends it back to testD3_orderFilled. The remaining four asserts (lines 22-26) do not fire, and the routine gets an 'x' tag.

Listing Thirty-two

            
class OrderTest(unittest.TestCase):

# see Listing Thirty-one...

@patch('__main__.Warehouse')
@unittest.expectedFailure
def testD3_orderFilled(self, testMock):
	"""Modified for failure"""
	# creating a test order
	testOrder = Order("mushrooms", 10)
	
	# configure the mock
	testMock.hasInventory.return_value = True
	testMock.getInventory.return_value = 10
	testMock.getInventory.side_effect = LookupError
	
	# perform the test
	testOrder.fill(testMock)
	print testOrder.isFilled()
	
	# perform the checks
	self.assertTrue(testOrder.isFilled())
	self.assertGreaterThan(testOrder._orderFilled, -1)
	
	testMock.hasInventory.assert_called_once_with("mushrooms")
	testMock.getInventory.assert_called_with("mushrooms", 10)

# see demo test script...

Both testD2_orderFilled and testD3_orderFilled use core patch as a function decorator.

Conclusion

Failure testing is integral to a unit testing. It helps reveal how a test subject reacts to a failure, and it helps isolate potential points of failures in said subject. And so, remember not only to test for success, but for failures too.

Recommended Reading

Martin Fowler. Mocks aren't Stubs

Michael Foord. Mock Documentation, Release 1.0.1 [PDF]

Mark Pilgrim. Testing for Failure, Dive into Python.

Python Software Foundation. Built-in Exceptions. Python 2.7.5 Documentation.

Python Software Foundation. Errors and Exceptions. Python 2.7.5 Documentation.

Python Software Foundation. unittest--Unit testing framework Python 2.7.5 Documentation.


José R.C. Cruz is a freelance technical writer based in British Columbia. He frequently contributes articles to MacTech, REALStudio Developer, and Dr. Dobb's.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.