Channels ▼
RSS

Design

Testing for Failures with Python


Failures in a Test Case

Next, you can control how a test routine reacts to a failure. To start, you can mark some routines as possible failures. When said routine fails, it is behaving as expected and gets an 'x' tag. When the routine "succeeds," it is suspect, earning itself a 'u' tag.

To mark a test routine, precede the routine with the decorator expectedFailure. Listing Eleven demonstrates this decorator in action. My test case BarTest has one routine test_bar (lines 5-16). The routine creates the mock object mockFoo and sets its side_effect attribute to ReferenceError (lines 9-10). Then it creates the test subject testBar and passes mockFoo to the subject's doBar() method (lines 13,16).

Listing Eleven

class BarTest(unittest.TestCase):

"""This test routine is supposed to fail"""
@unittest.expectedFailure
def test_bar(self):
	print "Running the test..."
	
	# prepare the test resource
	mockFoo = Mock(spec = Foo)
	mockFoo.side_effect = ReferenceError
	
	# prepare the test subject
	testBar = Bar()
	
	# run a test
	testBar.doBar(mockFoo)
# Results:
# failure: <type 'exceptions.ReferenceError'>
# failure: 
# failure: <traceback object at 0x46c7b0>
# doBar_:clean-up
    
# run the test
if __name__ == '__main__':
    unittest.main()
# Results:
# x
# OK (expected failures=1)

Here again, doBar() behaved as expected, catching ReferenceError with the third except handler, then sending it back to test_bar. Because test_bar does not have its own try…except block, it fails and gets tagged with an 'x'. In turn, BarTest increments the expected failures count by 1.

In Listing Twelve is a different example. Now I set the same side_effect attribute to MemoryError (line 10). When I pass mockFoo to doBar(), doBar() catches the failure signal with the second except handler. But doBar() does not send the signal back to test_bar. As a result, test_bar does not fail, it gets a tag of 'u', and BarTest increments the unexpected success count to 1.

Listing Twelve

class BarTest(unittest.TestCase):

"""This test routine is supposed to fail"""
@unittest.expectedFailure
def test_bar(self):
	print "Running the test..."
	
	# prepare the test resource
	mockFoo = Mock(spec = Foo)
	mockFoo.side_effect = MemoryError
	
	# prepare the test subject
	testBar = Bar()
	
	# run a test
	testBar.doBar(mockFoo)
# Results:
# failure:out-of-memory
# doBar_:clean-up
    
# run the test
if __name__ == '__main__':
    unittest.main()
# Results:
# u
# OK (unexpected successes=1)

You can also make a test routine fail by calling the TestCase method fail(). An explicit failure means the routine has had a problem, one unrelated to the test subject or resource. The failed routine gets an 'F' tag (note case) and the test case increments the failures count by 1.

Consider the test case in Listing Thirteen.

Listing Thirteen

  class BarTest(unittest.TestCase):
    """This test routine will fail"""
    def test_barA(self):
        print "Running the test (A)..."
        
        fooTest = "Peter Piper picked a peck of pickled peppers."
        fooTest = fooTest.rsplit(",")
        if (len(fooTest) > 1):
            # continue with the test
            pass
        else:
            self.fail()
# Results
# FAIL: test_barA (__main__.BarTest)
    
    """This test routine is suppose to fail"""
    @unittest.expectedFailure
    def test_barB(self):
        print "Running the test (B)..."
        
        # prepare the test resource
        mockFoo = Mock(spec = Foo)
        mockFoo.side_effect = ReferenceError
        
        # prepare the test subject
        testBar = Bar()
        
        # run a test
        testBar.doBar(mockFoo)
# Results
# failure: <type 'exceptions.ReferenceError'>
# failure: 
# failure: <traceback object at 0x46aaf8>
# doBar_:clean-up
    
# run the test
if __name__ == '__main__':
    unittest.main()
# Results
# Fx
# FAILED (failures=1, expected failures=1)

In test_barA, I have the rsplit() method to convert the text object fooTest into a list of words (lines 6-7). If the list has more than one word, the test continues. If not, test_barA invokes the method fail() (line 12). As for test_barB, I used the same test code from Listing Eleven.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.