Channels ▼
RSS

Code Coverage Analysis in Perl


Dr. Dobb's Journal

Is all of your Perl code actually executing? Are some statements invoked as expected? The Devel::Coverage module can give you the answer, but you've got to ask the right questions.

Perl comes with a built-in debugging mechanism that allows you to create your own debugger or use one created by someone else. Perl coders can also use the href="http://search.cpan.org/search?dist=Devel-Coverage">Devel::Coverage debugging module to determine just how much of a program actually executes. This testing process is called href="http://www.bullseye.com/coverage.html">coverage analysis. The Devel::Coverage module is available on the Comprehensive Perl Archive Network (CPAN). I'll show you how to use this sort of analysis to actually test all parts of your target program.

You use coverage analysis while testing your code (and everyone tests their code, right?) to ensure that you have done everything that necessary to exercise as much of the code as possible. Proper coverage analysis by itself does not make your code better or your programs bulletproof, but allows you to focus on problems in your code or in the test cases by examining different coverage metrics.

You can use several different coverage analysis metrics in your testing, including statement, condition, and path coverage, which respectively measure the degree to which your program executes all executable statements, tests each possible combination of logical conditions, or follows each possible path. You measure each of these differently, and may use different tools for each. href="http://search.cpan.org/search?dist=Devel-Coverage">Devel::Coverage measures statement coverage.

Suppose you had a program that defined a logical condition that was always

true, so that only one branch of the logical construct was ever followed. A

program with such an error might look like this:

	

	  1: #!/usr/bin/perl
	  2:
	  3: print "Hi there!\n";
	  4:
	  5: if( 1 )
	  6:       {
	  7:       print "I am where I should be\n"; 
	  8:       }
	  9: else
	 10:       {
	 11:       print "How did I get here?\n";
	 12:       }

	

Coverage analysis can spotlight this sort of error. Once you know it is there

you can fix it.

However, as I was writing this article and conducting my

module tests, I encountered a more common situation in which there is not a programming

error but an omission in testing. In the constructor for this module, I attempted

to open a file and process it. If I could not open the file I handled the

error.

	

        if( open FILE, $file )
                {
                # process the file
                }
        else
                {
                $errmsg = $!;
                return;
                }

 	

I do not have a programming error like I did in the last example, but in the tests that I created to exercise this piece of code I neglected to add a test that including trying to open an unreadable file. When I ran a coverage tool on my test program I discovered that I never executed the lines in the else block, which means that I never properly tested it and that I missed potential errors and problems with that code. Thus, I had distributed code to my colleagues that had never been executed. I think we all know how that story usually turns out.

You can do this sort of statement coverage analysis in Perl very easily using

some of the same principles that I discussed in href="http://www.ddj.com/columns/perl/2001/0104pl001/0104pl001.htm">an earlier

article about profiling code. In that article I discussed counting the

number of times that a particular line of source was executed when you ran a

program and how you could see that count. Statement coverage analysis differs in

that it shows you a count for every line with an executable statement, even if

that count is zero. Those are the ones that you need to examine further.

The href="http://search.cpan.org/search?dist=Devel-Coverage">Devel::Coverage module is a perl debugger that you can invoke by using the href="http://www.perldoc.com/perl5.6/pod/perlrun.html">-d switch with perl and the name of a module in the href="http://search.cpan.org/search?mode=module&query=Devel">Devel::* namespace.

	

	perl -d:Coverage script.pl

	

Perl runs the script through the href="http://search.cpan.org/search?dist=Devel-Coverage">Devel::Coverage debugger and saves the results in a file named after the script, but with .cvp appended to the name. Inside this file href="http://search.cpan.org/search?dist=Devel-Coverage">Devel::Coverage stores a data structure that you can read yourself if you are the sort who likes the Obfuscated Perl Contest, or you can use the coverperl utility, which comes with the module and pretty prints the results to standard output.

	

	coverperl script.pl.cvp

	

I look at the results in script.pl.cvp with coverperl.

	

	Total of 1 instrumentation runs.

	/usr/home/brian/DDJ/Coverage/script.pl
	        1    line 3   
	        1    line 5   
	        1    line 7   
	        0    line 11

	

Notice that line 11 was not reached. The coverage was incomplete, which means I need to pay attention to what I did not execute either because it was unreachable code, which can be removed, or because I did not provide an adequate test. Notice that this only tests that each statement was encountered and executed — not that the program did what it should have done.

This debugger can also keep the results of many runs of the same program. If I leave the script.pl.cvp file as is and re-run the script under the debugger just like before, when I look at the results with coverperl I see the accumulation of results for both runs.

	

	Total of 2 instrumentation runs.

	/usr/home/brian/DDJ/Coverage/script.pl
	        2    line 3   
	        2    line 5   
	        2    line 7   
	        0    line 11 

	

Even though I might not execute all of the statements in one run of a

program, I might execute all of them during several runs with different input

and conditions. For instance, a program that tells me whether a number is odd

or even will only execute some of the lines when the number is odd and the

others when the number is even. If I do not reach some of the statements during

a single run of the program, I do not necessarily have an error.

	

	 1: #!/usr/bin/perl
	 2:
	 3: if( $ARGV[0] % 2 )
	 4:        {
	 5:        print "$ARGV[0] is odd\n";
	 6:        }
	 7: else
	 8:        {
	 9:        print "$ARGV[0] is even\n";
	10:        }

	

When I run this once, only one branch is executed.

	

	% perl -d:Coverage even.pl 8
	8 is even

	

The coverage is incomplete because I never executed line 5.

	

	Total of 1 instrumentation runs.

	/usr/home/brian/DDJ/Coverage/even.pl
	        1    line 3   
	        0    line 5   
	        1    line 9   

	

When I run the same program again with different input, chosen

judiciously to execute the other branch,

	

	% perl -d:Coverage even.pl 7
	7 is odd

	

I see that I have reached all of the lines.

	

	Total of 2 instrumentation runs.

	/usr/home/brian/DDJ/Coverage/even.pl
	        2    line 3   
	        1    line 5   
	        1    line 9   

	

Remember that just because I executed all of the lines of the program does

not mean I have determined that the program is without bugs or is correct.

How can you apply this to your Perl work? If you create Perl modules, and you start with href="http://www.perldoc.com/perl5.6/bin/h2xs5.003.html">h2xs or can pretend that you did, you also create a test.pl in which you can define various tests of your module. These tests can be run as part of the installation process of the module to test functionality and compatibility with the target system. If you are not familiar with this process you might want to take a look at href="http://www.perldoc.com/perl5.6/bin/h2xs5.003.html">h2xs to see what it does.

You can use the test.pl script to measure coverage without changing anything else in your Makefile. Normally you run test.pl by running make test in the module directory. To use the debugger you do the same thing that I did before but with a little twist. When you perform your initial make, the module and its associated parts are put into a build library in a directory named blib. If you want your test.pl to find your modules, which are not in the standard library locations yet, you need to tell perl where to look for them. You can do this in many ways, but the easiest, and least painful way in my opinion, is to specify the build library location with the href="http://www.perldoc.com/perl5.6/pod/perlrun.html">-I switch.

	

	% perl -d:Coverage -Iblib/lib test.pl

	

Everything happens as before. The href="http://search.cpan.org/search?dist=Devel-Coverage">Devel::Coverage modules puts the results of the analysis into test.pl.cvp, which you can then examine with coverperl. If you have created a good test.pl, coverperl will tell you that every line was executed. If you find some lines that are not executed, you add more tests to make sure that they are just like I did in my example at the beginning of this article.

Now that you have an easy way to

measure how much of your modules you actually test, try it on some of your test

scripts and find out where you need to add tests. If you find that you covered all

of your code, pat yourself on the back.  


brian d foy has been a Perl user since 1994. He is founder of the first Perl users group, href="http://ny.pm.org/">NY.pm, and Perl Mongers, the Perl advocacy organization. He has been teaching Perl through Stonehenge Consulting for the past three years, and has been a featured speaker at The Perl Conference, Perl University, YAPC, COMDEX, and Builder.com. Some of brian's other articles have appeared in The Perl Journal.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video