Channels ▼
RSS

Web Development

Using Perl to Stress-Test Server Applications


Nov02: Using Perl to Stress-Test Server Applications

Moshe is a systems administrator and operating-system researcher and has an M.S. and a Ph.D. in computer science. He can be contacted at moshe@moelabs.com.


It is one of the ironies of a programmer's life that when we've finished writing a program, the real work has just begun. Once the program compiles cleanly and seems to do what it was supposed to do, good programmers then need to find out how fast (or slow) the program performs its designed purpose. In other words, we need to stress-test the program.

There are commercial stress-test applications in the market, but they are expensive for individuals or small development firms, and often too generalized—capable of most everything, but not really optimized for your particular situation. With its myriad modules (I call them "connectors"), Perl is a great tool to help you stress-test your particular program.

As a senior contributing editor at BYTE.com, I've been able to compare operating systems to one another (usually Linux against something else) over the last few years, and the way I do it is by running the exact same stress-test against standard applications like web servers and databases (MySQL). As it turns out, Perl is my tool of choice for this purpose, thanks in part to the CPAN repository of modules.

Parallel Fork Manager

One of the modules that best suits the stress-test purpose is the Parallel Fork Manager (http://aspn.activestate.com/ASPN/CodeDoc/ Parallel-ForkManager/ForkManager.html). Parallelizable parts of a Perl program can be forked into separate processes that continue to run concurrently with the parent process in a multiCPU environment. In a massively parallel environment, such as my own openMosix technology (http://www.openMosix.org/), every forked Perl program always finds a CPU to run on. The total run time is therefore drastically reduced. The Perl program in Listing 1 can run from many nodes in parallel in an openMosix cluster, and the server being tested is submitted to a veritable bombardment of client-side requests.

Using the Perl Fork Manager is easy. A simple construct like Example 1 parallelizes very nicely—and by using openMosix you don't even have to care about distribution in the cluster. Several callbacks (or event handlers) are possible, which are called on certain events. Among the callbacks, you typically use run_on_start $code, which defines a subroutine called when a freshly forked child begins execution.

Looking at Listing 1, all the script does is run $processes in parallel and create tables with $rows, then starts $transactions (SQL queries) against the database. In the beginning, you define the various fork and database connection managers. Then, you define an array of names that will be used later to populate the database with sample records. If it is important to increase the cardinality of the indices, then you can do so by building an array of n names by reading in n words from the UNIX dict file (which in Red Hat is in /usr/share/dict/linux.words).

Using the well-known DBI module, you then connect to the database (Oracle in this example) and first drop, then create the tables of this fictitious application. In Part Four of the code (see comments), you tell the openMosix clustering system (through the convenient /proc filesystem of Linux) to make sure all forked Perl programs are free to migrate to other nodes in the cluster to evenly distribute them and keep the cluster load-balanced. Part Seven is where you parallelize the SQL queries to the database using the Fork Manager module.

That's it. The entire program consists mostly of the SQL handling for connection to the database and housekeeping. The code related to the Fork Manager is just a few lines. The same program structure can be used for stress-testing any server application. Depending on what you need to do with the results, you can use various Perl modules to represent the results graphically or publish them onto a web page.

For my purposes, a simple run-time summary is enough and it keeps the program simple. Everybody is encouraged to use this Perl script. I'd be interested to know how you enhanced it for your particular needs.

TPJ

Listing 1

#! /usr/bin/perl -w

# Part One Modules and parameters
use Parallel::ForkManager;
use DBI;
srand;

############# configruation part #######################
my $processes=30;
my $transactions=5000;
my $rows=100000;  # rows to be added to the database
########################################################

my $p;

# Part Two - Seed for DB population@
NAMES = qw/Alf Ben Benny Daniel David Foo Bar Moshe 
                                        Avivit Jon Linus Larry Safety First/;

#Part Three - DB environment
my $username;
my $money;
my $id;
my $exists=0;
my $commandlineargs=0;

$SID = $ENV{ "ORACLE_SID" };

if (!$SID) {

printf("No ORACLE_SID environment!\n");
printf("... do not know to which database you want to connect\n");
printf("You have to export the database-name in the environment\n");
printf("variable ORACLE_SID e.g.\n");
printf("export ORACLE_SID [mydb]\n");

exit -1
}

# Part Four - openMosix related. unlocks this process and all its children
sub unlock { 

#open (OUTFILE,">/proc/self/lock") || 
#ie "Could not unlock myself!\n"; 
#print OUTFILE "0"; 

} 

unlock;

# Part Five - Here we connect to the DB and populate the tables with records
sub filldb {

my $dbh = DBI->connect( "dbi:Oracle:$SID",
                        "scott",
                        "tiger",
                        {
                          RaiseError => 1,
                          AutoCommit => 0
                        }
                      ) || die "Database connection not made: $DBI::errstr";
                    for ($loop=0; $loop<$rows; $loop++) {
                            # prepare the random values
                            srand;
                            $p=rand();
                            $username = $NAMES[rand(@NAMES)];
                            $money=rand()*(rand()*100);
                            $id=rand()*1000;
                            $id=sprintf("%d", $id);
                            # insert
                             my $sql2 = qq{ insert into stress 
                                           values ($id,'$username',$money) };
                             my $sth2 = $dbh->prepare( $sql2 );

                             $sth2->execute();
                             $sth2->finish();
            }
                    $dbh->disconnect();
}
# delete old entries from db
sub deldb {
my $dbh = DBI->connect( "dbi:Oracle:$SID",
                        "scott",
                        "tiger",
                        {
                          RaiseError => 1,
                          AutoCommit => 0
                        }
                      ) || die "Database connection not made: $DBI::errstr";

                        # drop
                        my $sql1 = qq{ select TABLE_NAME from user_tables 
                                                 where TABLE_NAME='STRESS' };
                        my $sth1 = $dbh->prepare( $sql1 );
                        $sth1->execute();

            # if the table does not exists
                        while( $sth1->fetch() ) {
             $exists=1;
                         }
            if (!$exists) {

             # create
                         my $sql2 = qq{ create table STRESS ( id number, 
                                         name varchar2(255), money float) };
                         my $sth2 = $dbh->prepare( $sql2 );

                         $sth2->execute();
                         $sth2->finish();

             } else {

                         # cleanup
                         my $sql3 = qq{ delete from STRESS };
                         my $sth3 = $dbh->prepare( $sql3 );

                         $sth3->execute();
                         $sth3->finish();
            }
            $sth1->finish();
                        $dbh->disconnect();
}

# Part Six - Stress-testing the DB with parallel instances 

sub stressdb {
my $dbh = DBI->connect( "dbi:Oracle:$SID",
                        "scott",
                        "tiger",
                        {
                          RaiseError => 1,
                          AutoCommit => 0
                        }
                      ) || die "Database connection not made: $DBI::errstr";
        for ($loop=0; $loop<$transactions; $loop++) {
            # prepare the random values
            srand;
            $p=rand();
            $username = $NAMES[rand(@NAMES)];
            $money=rand()*(rand()*100);

            $id=rand()*1000;
            $id=sprintf("%d", $id);

            # update transactioin
            if ($p<0.3) {

             printf("update\n");

             my $sql = qq{ update stress set name='$username', 
                                                money=$money where id=$id };
             my $sth = $dbh->prepare( $sql );

             $sth->execute();
             $sth->finish();
            }

            # select 
            if (($p>0.3) and ($p<0.7)) {
             printf("select\n");
             $sql = qq{ select id, name, money from stress };
             my $sth = $dbh->prepare( $sql );

             $sth->execute();
             my( $sid, $sname, $smoney );

             $sth->bind_columns( undef, \$sid, \$sname, \$smoney );
             while( $sth->fetch() ) {

              # print "$sid $sname $smoney\n";
             }
            }

            # delete + insert
            if ($p>0.7) {
             printf("delete/insert\n");

             # delete first
                         my $sql1 = qq{ delete from stress where id=$id };
                         my $sth1 = $dbh->prepare( $sql1 );

                         $sth1->execute();
             $sth1->finish();

            # insert
                         my $sql2 = qq{ insert into stress 
                                          values ($id,'$username',$money) };
                         my $sth2 = $dbh->prepare( $sql2 );

                         $sth2->execute();
                         $sth2->finish();

            }
        }
        $dbh->disconnect();
}

############## main ######################

print "\nStarting the orastress test\n\n";

# check for commandline arguments
foreach $parm (@ARGV) {
 $commandlineargs++;
}
$parm=0;
if ($commandlineargs!=3) {

 # get values from the user
 print "How many rows the database-table should have ? : ";

 $rows=<STDIN>;
 chomp($rows);

 while ($rows !~ /\d{1,10}/){
  print "Invalidate input! Please try again.\n";
  print "How many rows the database-table should have ? : ";
  $rows = <STDIN>;
  chomp($rows);
 }

 print "How many clients do you want to simulate ? : ";
 $processes=<STDIN>;
 chomp($processes);
 while ($processes !~ /\d{1,10}/){
  print "Invalidate input! Please try again.\n";
  print "How many clients do you want to simulate ? : ";

  $processes = <STDIN>;
  chomp($processes);
 }

 print "How many transactions per client do you want to simulate ? : ";
 $transactions=<STDIN>;
 chomp($transactions);

 while ($transactions !~ /\d{1,10}/){
  print "Invalidate input! Please try again.\n";
  print "How many transactions per client do you want to simulate ? : ";

  $transactions = <STDIN>;
  chomp($transactions);
 }

 } else {
 # parse the values from the command line

 $rows = $ARGV[0];
 $processes = $ARGV[1];
 $transactions = $ARGV[2];

 if (($rows !~ /\d{1,10}/) or ($processes !~ /\d{1,10}/) or 
                                            ($transactions !~ /\d{1,10}/)) {
  print ("Invalidate input! Please try again.\n");
  print ("e.g.   stress-oracle.pl 10 10 10\n");

  exit -1;

  } else {
  print("got the values for the stress test from the command line\n");
  print("rows = $rows\n");

  print("processes = $processes\n");
  print("transactions = $transactions\n");

 }
}

# cleaning up old db
print("delete old entries from db\n");

deldb();

print("fill db with $rows rows\n");

filldb();

# Part Seven
print("starting $processes processes\n");
my $pm = new Parallel::ForkManager($processes);

$pm->run_on_start(
  sub { my ($pid,$ident)=@_;
  print "started, pid: $pid\n";
  }
);

for ($forks=0; $forks<$processes; $forks++){
  $pm->start and next;
  stressdb;

  $pm->finish;
}

$pm->wait_all_children;
$pm->finish;

print "Simulation finished\n";


Back to Article


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 
Dr. Dobb's TV