Channels ▼
RSS

Design

Testing for Failures with Python


When I run BarTest, I get two sets of test results. The first set tells me that test_barA failed, tagging it with an 'F' and showing a failure count of 1. Then the second set tells me that the doBar() method failed (as expected), tagging it with an 'x' and showing an expected failures count of 1.

Now look at Listing Fourteen. Here, I gave test_bar its own private function, barDivi() (lines 6-21). When the second argument (barY), a tuple, has only zeros, barDivi() calls fail(), passing along a failure message (lines 7-8). When I run BarTest, not only do I see test_bar having failed, I also see the failure message supplied by barDivi().

Listing Fourteen

class BarTest(unittest.TestCase):
    """This test routine will fail"""
    def test_bar(self):
        
        # define a private function
        def barDivi(barX, barY):
            if (barY[0] == 0 and barY[1] == 0):
                self.fail("Invalid second argument")
            else:
                barLCD = barY[0]**2
                barLCD += barY[1]**2
                
                barRe = barX[0] * barY[0]
                barRe += barX[1] * barY[1]
                barRe /= barLCD
                
                barIm = barX[1] * barY[0]
                barIm -= barX[0] * barY[1]
                barIm /= barLCD
                
                return (barRe, barIm)
        
        print "Running the test (A)..."
        
        barX = 3,4
        barY = 0,0
        barZ = barDivi(barX, barY)
# Results
# AssertionError:Invalid second argument
# FAIL: test_bar (__main__.BarTest)
    
# run the test
if __name__ == '__main__':
    unittest.main()
# Results
# F
# FAILED (failures=1)

Finally, you can control how the test case reports a failure. The property failureException returns the most recent failure. It defaults to AssertionError to indicate a failed assert. You can, however, assign failureException a different value to reflect the current failure.

The property longMessage changes the failure message. Set it to True and the resulting failure message consists of the arguments, the assert type, plus any custom message. Set longMessage to False (default), and the failure message consists only of the custom message.

Listing Fifteen demonstrates the effects of these two properties. The routines test_barA and test_barB are similar to the ones from Listing Thirteen. But, in test_barA, I used assertGreater to check the word count produced by rsplit() (line 24). In test_barB, I wrapped the test code in a try…except block and I set failureException to sys.exc_info() (line 42).

Listing Fifteen

class BarTest(unittest.TestCase):
    
    # preparing to test
    def setUp(self):
        """ Setting up for the test """
        self.longMessage = True
# Result:
# AssertionError: 1 not greater than 1 : Incorrect word count

    # ending the test
    def tearDown(self):
        """Cleaning up after the test"""
        print self.failureException
        print
# Result:
# (<type 'exceptions.ReferenceError'>, ReferenceError(), <traceback object at 0x469aa8>)
    
    """This test routine will fail"""
    def test_barA(self):
        print "Running the test (A)..."
        
        fooTest = "Peter Piper picked a peck of pickled peppers."
        fooTest = fooTest.rsplit(",")
        self.assertGreater(len(fooTest), 1, "Incorrect word count")
        
    """This test routine is suppose to fail"""
    @unittest.expectedFailure
    def test_barB(self):
        print "Running the test (B)..."
        
        # prepare the test resource
        try:
			mockFoo = Mock(spec = Foo)
			mockFoo.side_effect = ReferenceError
			
			# prepare the test subject
			testBar = Bar()
			
			# run a test
			testBar.doBar(mockFoo)
        except:
        	self.failureException = sys.exc_info()
    
# run the test
if __name__ == '__main__':
    unittest.main()
# Result:
# Fu
# FAIL: test_barA (__main__.BarTest)
# FAILED (failures=1, unexpected successes=1)

Now in the setUp() method, I set longMessage to True (lines 4-6). When test_barA fails, its failure message will read:

AssertionError: 1 not greater than 1 : Incorrect word count

But if I set longMessage to False, the failure message from test_barA will read:

AssertionError: Incorrect word count

I also have the tearDown() method print the value of failureException to stdout (lines 11-14). When test_barB fails, I get a tuple giving the failure type and description, and a traceback object (line 16).

Asserting for Failures

The TestCase class has two asserts that work with failure signals. The first assert, assertRaises, takes one argument — the exception — as the expected failure. The second assert, assertRaisesRegexp takes two arguments: the expected failure and the text pattern matching the failure message. Both asserts fire when the signal does not match the one expected.

assertRaises(failure-signal)
assertRaises(failure-signal, text-pattern)

Listing Sixteen demonstrates assertRaises at work. In test_bar, I set the side_effect attribute for mockFoo to MemoryError (line 8). When I pass mockFoo to the instance method doBar(), doBar()'s second except handler catches the failure signal. But the handler did not send the signal back to test_bar, causing assertRaises to fire.

Listing Sixteen

class BarTest(unittest.TestCase):
    
    def test_bar(self):
        print "Running the test..."
        
        # prepare the test resource
        mockFoo = Mock(spec = Foo)
        mockFoo.side_effect = MemoryError
        
        # prepare the test subject
        testBar = Bar()
        
        # run a test
        with self.assertRaises(MemoryError) as fooErr:
            testBar.doBar(mockFoo)
# Result:
# failure:out-of-memory
# doBar_:clean-up
    
# run the test
if __name__ == '__main__':
    unittest.main()
# Result:
# AssertionError:MemoryError not raised
# F
# FAIL: test_bar (__main__.BarTest)
# FAILED (failures=1)

Listing Seventeen shows another example. This time, I set mockFoo's side_effect attribute to ReferenceError (line 8). I pass mockFoo to doBar(), and doBar() catches the signal with the third except handler. The handler sends the signal back to test_bar, causing assertRaises not to fire.

Listing Seventeen

  class BarTest(unittest.TestCase):
  
    def test_bar(self):
        print "Running the test..."
        
        # prepare the test resource
        mockFoo = Mock(spec = Foo)
        mockFoo.side_effect = ReferenceError
        
        # prepare the test subject
        testBar = Bar()
        
        # run a test
        with self.assertRaises(ReferenceError) as fooErr:
            testBar.doBar(mockFoo)
# Result:
# failure: <type 'exceptions.ReferenceError'>
# failure: 
# failure: <traceback object at 0x46a940>
# doBar_:clean-up
    
# run the test
if __name__ == '__main__':
    unittest.main()

Listing Eighteen demonstrates assertRaisesRegexp. When I set the side_effect attribute to ReferenceError, I also assigned it with a failure message (line 8). To the assert, I pass the same ReferenceError signal and I pass the text pattern "\Amissing" (line 14). I run the test and found doBar() catching ReferenceError with its third except handler. That same handler sends ReferenceError back to test_bar. The assert does not fire, because the failure matches the assert conditions.

Listing Eighteen

class BarTest(unittest.TestCase):

    def test_bar(self):
        print "Running the test..."
        
        # prepare the test resource
        mockFoo = Mock(spec = Foo)
        mockFoo.side_effect = ReferenceError("missing object reference")
        
        # prepare the test subject
        testBar = Bar()
        
        # run a test
        with self.assertRaisesRegexp(ReferenceError, "\Amissing") as fooErr:
            testBar.doBar(mockFoo)
# Result:
# failure: <type 'exceptions.ReferenceError'>
# failure: missing object reference
# failure: <traceback object at 0x46c940>
# doBar_:clean-up
    
# run the test
if __name__ == '__main__':
    unittest.main()


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.