Channels ▼
RSS

Web Development

Better Documentation Through Testing


Nov02: Better Documentation Through Testing

brian has been a Perl user since 1994. He is founder of the first Perl Users Group, NY.pm, and Perl Mongers, the Perl advocacy organization. He has been teaching Perl through Stonehenge Consulting for the past five years, and has been a featured speaker at The Perl Conference, Perl University, YAPC, COMDEX, and Builder.com.


I document all of my modules and scripts, and although I readily find out about program errors because people are quick to tell me their script either fails to run or does the wrong thing, almost nobody reports problems with the documentation.

Perl has a basic document format called "Plain Old Documentation," or POD. The Perl documentation reader, perldoc, follows the adage, "Be strict in what you output, and liberal in what you accept." It takes most of the bad POD formatting that I give it and turns it into something the user can read. If I make errors in my POD markup, perldoc often silently fixes them. I have the same problem with HTML. I can write bad HTML that the browser fixes so it can display it.

Perl comes with a POD validator, podchecker, based on the Pod::Checker module, which is part of the Perl Standard Library. Given a file with POD directives, podchecker tells me where I messed up. I tell podchecker which file (not which module, like perldoc) to check and it shows me the POD errors and questionable constructs; see Listing 1.

I am lazy, in the Perly sense of the word, so I want this to happen automatically. As I add new features to modules or move code around, I change the documentation. I may add more documentation, get rid of old explanations, or rearrange it. Anywhere in this process I might introduce a POD error. I do not want to use podchecker every time I make a change.

I could check the documentation just before I release a new version, but I usually forget to do that. I have automated most of the steps to release something to CPAN, so I should automate checking the documentation format as well.

Test::Pod

I wrote the Test::Pod module to automatically check my modules and scripts for POD errors. It does not check if I documented everything that I should have, like Pod::Coverage does. It simply wraps Pod::Checker, which only checks the POD markup, in a Test::Builder interface and calls it pod_ok().

Once I have my pod_ok function, which I show in gory detail later, I can use it in a test script. This snippet uses Test::More, the basis of many new test files. Test::More comes with the latest stable version of Perl, 5.8.0, which also comes with Test::Tutorial, which explains the basics of testing.

# t/pod.t
use Test::More tests => 1;
use Test::Pod;
pod_ok( 'blib/lib/ISBN.pm' );

The test fails if Pod::Checker finds a POD error in the ISBN.pm file from my Business::ISBN distribution.

Most of my distributions have more than one module in it, though. Andy Lester showed me a cool hack to test all of the modules without remembering which modules I had. If I add a new module to the distribution, this test finds it automatically.

# t/pod.t
BEGIN {
    use File::Find::Rule;
    @files = File::Find::Rule->file()->name( 
        '*.pm' )->in( 'blib/lib' );
    }

use Test::More tests => scalar @files;
use Test::Pod;

foreach my $file ( @files )
    {
    pod_ok( $file );
    }

The File::Find::Rule module allows me to find all of the modules in the named directory and the build library ('blib/lib' in this case) very easily. Once I have all the filenames that I want to test in @files, I simply loop through @files to test each one.

I run this test like any other test file. Once I create the module Makefile from Makefile.PL, I run make test, which tells Test::Harness to do its magic. It runs all of the *.t files in the t directory, collects the results, and reports what it finds; see Listing 2.

If a test fails, the Test::Harness report is different. In this case, I edited the ISBN.pm file to remove an =over POD directive so the POD now has an error. Test::Pod reports that it found an error at line 400. Pod::Checker correctly identifies the problem as a missing =over. Test::Harness prints a summary of the failed tests at the end; see Listing 3.

Creating the Test Module

Test::Builder is the brainchild of Michael Schwern and chromatic, and in my opinion, it's the best thing to happen to Perl in years. This module allows other people to plug into Test::Harness with their own specialized modules. The Comprehensive Perl Archive Network (CPAN) has about 20 specialized Test modules so far.

These specialized modules have the same advantages as any other module—I can write a test function once and use it over and over again. The function standardizes the way I do things and moves more code out of the test files. I take any chance I can get to move code out of the test files. More code means more points of failure. I have to try really hard to mess up pod_ok().

The Test::Builder interface is very simple. In my specialized test module, I create a Test::Builder object, which is a singleton, so all of the specialized test scripts play well together.

my $Test = Test::Builder->new();

I create a test function, called pod_ok in Test::Pod, that tells Test::Builder if the test succeeded or failed, and optionally outputs some error information. Test::Builder's ok() method handles the result. If I think the test passed, I give ok() a true value, and a false value otherwise. The meat of pod_ok() is very simple—tell Test::Builder the test either passed or failed.

sub pod_ok
    {
    $pod_ok = _check_pod();
    if( $pod_ok ) { $Test->ok(1) }
    else          { $Test->ok(0) }
    }

Everything else in the function supports those simple statements in the pod_ok function. Test::Builder takes care of the rest.

To actually test the POD, I created an internal function, _check_pod, in Test::Pod; see Listing 4. I already defined the constants NO_FILE, NO_POD, ERRORS, WARNINGS, and OK, which described the conditions that Pod::Checker can report. The function returns an anonymous hash. The result is the value of the result key, error messages are the value for the output key, the number of errors is the value of the errors key, and the number of warnings is the value for the warnings key. The rest of the code puts the right things in the hash.

The first line in the subroutine takes a filename off of the argument stack, and the third line checks the file's existence. If the file does not exist, the function returns an anonymous hash with the result NO_FILE.

The next bit of code ties the variable $hash{output} to IO::Scalar. Pod::Checker can write messages to a file handle, and I want to intercept that output. It shows up in $hash{output} instead of the terminal.

I get a POD checker object from Pod::Checker, and then parse the specified file. Pod::Checker puts all output in $hash{output}. The do block puts the numbers of errors and warnings in the right keys, then returns the right constant for the condition that Pod::Checker reported.

Finally, _check_pod returns the anonymous hash that the pod_ok function can use to tell Test::Builder what happened. The pod_ok() function (see Listing 5) does the same thing it did before, although it has to do a little more work to decide if the test passed or not.

The first line takes a file name off of the argument list, and then passes that to _check_pod. The second argument specifies my expected result and default to OK. The third argument gives a name to the test that I can see in the verbose output of Test::Harness. It defaults to a message that includes the name of the file.

The rest of the function is a series of if-elsif statements, with one test for each possible condition. I can specify an expected result in the second argument. If Pod::Checker finds the condition I expected, then the test succeeds, and fails otherwise. If I do not specify an expected condition, pod_ok assumes I only want the test to pass if Pod::Checker finds neither error nor warnings.

Every branch of the if-elsif structure calls Test::Builder's ok() method. If that branch represents a success, pod_ok passes 1 to ok(), and 0 otherwise.

If pod_ok fails a test, it also uses Test::Builder's diag() method to give an error message.

Conclusion

Testing my work is simple if I use Test::Builder. I can create specialized test modules to check all sorts of things other than normal script execution. My modules have better-formatted documentation because Test::Pod automatically tells me about problems.

TPJ

Listing 1

podchecker /usr/local/lib/perl5/site_perl/darwin/DBI.pm

*** WARNING: (section) in 'perl(1)' deprecated at line 4926 in file
   /usr/local/lib/perl5/site_perl/darwin/DBI.pm
*** WARNING: (section) in 'perlmod(1)' deprecated at line 4926 in file
   /usr/local/lib/perl5/site_perl/darwin/DBI.pm
*** WARNING: (section) in 'perlbook(1)' deprecated at line 4926 in file
   /usr/local/lib/perl5/site_perl/darwin/DBI.pm
*** ERROR: unresolved internal link 'bind_column' at line 1819 in file
   /usr/local/lib/perl5/site_perl/darwin/DBI.pm
*** WARNING: multiple occurence of link target 'trace' at line - in file
   /usr/local/lib/perl5/site_perl/darwin/DBI.pm
*** WARNING: multiple occurence of link target
   'Statement (string, read-only)' at line - in file
   /usr/local/lib/perl5/site_perl/darwin/DBI.pm
/usr/local/lib/perl5/site_perl/darwin/DBI.pm has 1 pod syntax error.

Back to Article

Listing 2

localhost_brian[3150]$ make test
cp ISBN.pm blib/lib/Business/ISBN.pm
cp Data.pm blib/lib/Business/ISBN/Data.pm
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e"
   "test_harness(0, 'blib/lib', 'blib/arch')" t/load.t t/pod.t t/isbn.t
t/load....ok                                                                 
t/pod.....ok                                                                 
t/isbn....ok 20/21                                                           
Checking ISBNs... (this may take a bit)
t/isbn....ok                                                                 
All tests successful.
Files=3, Tests=25, 358 wallclock secs (116.90 cusr +  2.96 csys = 119.86 CPU)

Back to Article

Listing 3

localhost_brian[3152]$ make test
cp ISBN.pm blib/lib/Business/ISBN.pm
Skip blib/lib/Business/ISBN/Data.pm (unchanged)
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e"
   "test_harness(0, 'blib/lib', 'blib/arch')" t/load.t t/pod.t t/isbn.t
t/load....ok                                                                 
t/pod.....NOK 1#     Failed test (t/pod.t at line 12)                        
# Pod had errors in [blib/lib/Business/ISBN.pm]
# *** ERROR: =item without previous =over at line 400 in file
   blib/lib/Business/ISBN.pm
# blib/lib/Business/ISBN.pm has 1 pod syntax error.
t/pod.....ok 2/2# Looks like you failed 1 tests of 2.                       
t/pod.....dubious                                                            
        Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 1
        Failed 1/2 tests, 50.00% okay
t/isbn....ok 20/21                                                           
Checking ISBNs... (this may take a bit)
t/isbn....ok                                                                 
Failed Test Stat Wstat Total Fail  Failed  List of Failed
----------------------------------------------------------------------------
t/pod.t        1   256     2    1  50.00%  1
Failed 1/3 test scripts, 66.67% okay. 1/25 subtests failed, 96.00% okay.
make: *** [test_dynamic] Error 35

Back to Article

Listing 4

sub _check_pod
    {
    my $file = shift;
    
    return { result => NO_FILE } unless -e $file;

    my %hash    = ();
    my $output;
    $hash{output} = \$output;
    
    my $checker = Pod::Checker->new();
    
    # i pass it a tied filehandle because i need to fool
    # Pod::Checker into thinking it is sending the errors
    # somewhere so it will count them for me.
    tie( *OUTPUT, 'IO::Scalar', $hash{output} );    
    $checker->parse_from_file( $file, \*OUTPUT);
        
    $hash{ result } = do {
        $hash{errors}   = $checker->num_errors;
        $hash{warnings} = $checker->can('num_warnings') ?
            $checker->num_warnings : 0;
        
           if( $hash{errors} == -1  ) { NO_POD   }
        elsif( $hash{errors}   > 0  ) { ERRORS   }
        elsif( $hash{warnings} > 0  ) { WARNINGS }
        else                          { OK }
        };
    
    return \%hash;
    }

Back to Article

Listing 5

sub pod_ok
    {
    my $file     = shift;
    my $expected = shift || OK;
    my $name     = shift || "POD test for $file";
    
    my $hash = _check_pod( $file );
            
    my $status = $hash->{result};
    
    if( defined $expected and $expected eq $status )
        {
        $Test->ok( 1, $name );
        }
    elsif( $status == NO_FILE )
        {
        $Test->ok( 0, $name );
        $Test->diag( "Did not find [$file]" );
        }
    elsif( $status == OK )
        {
        $Test->ok( 1, $name );
        }
    elsif( $status == ERRORS )
        {
        $Test->ok( 0, $name );
        $Test->diag( "Pod had errors in [$file]\n",
            ${$hash->{output}} );
        }
    elsif( $status == WARNINGS and $expected == ERRORS )
        {
        $Test->ok( 1, $name );
        }
    elsif( $status == WARNINGS )
        {
        $Test->ok( 0, $name );
        $Test->diag( "Pod had warnings in [$file]\n",
            ${$hash->{output}} );
        }
    elsif( $status == NO_POD )
        {
        $Test->ok( 0, $name );
        $Test->diag( "Found no pod in [$file]" );
        }
    else
        {
        $Test->ok( 0, $name );
        $Test->diag( "Mysterious failure for [$file]" );
        }
    }

Back to Article


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 
Dr. Dobb's TV