Testbots let you move the effort of building and testing from your shoulders to that of software robots.
April 11, 2006
URL:http://www.drdobbs.com/tools/i-testbot/185300349
Ralf is principal of PERA Software Solutions and can be contacted at [email protected].
There was a time when humanity faced the universe alone and without a friend. Now he has creatures to help him; stronger creatures than himself more faithful, more useful, and absolutely devoted to him. Mankind is no longer alone.
Isaac Asimov, I, Robot
Almost all software engineering experts agree that continuous integration is superior to the "big-bang" integration approach employed in the dark ages. Especially agile methodologies such as Extreme Programming (XP) promote development processes that let developers add functionality little-by-little, thereby significantly reducing project risk.
To ensure that frequent code check-ins (so-called "code deliveries," or "deliveries," for short) are of high quality, frequent build and test cycles are inevitable. One such approach is the now famous "Daily Build and Smoke Test" process, described in detail in Steve McConnell's book Rapid Development (Microsoft Press, 1996). While daily building and smoke testing significantly reduces project risk, for large and complex projects you may want to take this approach to extremes by validating your code base on a check-in basisthat is, many times a day. The advantages are obvious. If you check in buggy code, it will be immediately detected, which means that there will be no more time-consuming "Who's broken the build?" quests.
In this article, I present an approach that puts the Herculean building and testing effort on the shoulders of a test robot, or "testbot."
One of my current projects is a complex embedded-systems project. It isn't particularly large (around 20 developers and 100 KLOC), but it is a multiplatform, multifeature kind of project. At any timeor more precisely, within a rather short period of timethe system has to be in a releasable state for any hardware platform and various combinations of features supported by our common code base. Here, I refer to such a combination of features on a particular hardware platform as a "target."
Working on this project used to be like walking on a mine field. If you added a feature or fixed a bug for some target, there was a certain probability that you broke the build on another target or at least introduced a compiler warning. Over time, these issues tended to accumulate, resulting in a downward spiral. Lots of effort had to be expended later on to get back to a clean build.
Obviously, requiring the developers to verify all of the other platforms and products before delivering their changes would have taken way too much time, let alone the fact that not every developer had access to all of the toolchains (compilers, linkers, in-circuit emulators, and the like) that the code base supports.
I solved this problem by assigning dedicated testbots to all the targets we support. The testbots are workstations that host a program that listens on the source-code-versioning system (ClearCase, in our case) for new deliveries of developers. When a developer finishes delivering her task to the integration branch, the testbot executes the following steps:
The testbot software was quite easy to implement. In total, it comprises less than 1000 lines of Perl code; however, the size depends on the kind of source-code-versioning system you are usingdifferent systems offer different solutions for parallel software development and querying of check-in information. Listing One is the testbot pseudocode implementation. For simplicity, I've left out logging and error handling.
loop forever sleep for 5 minutes check versioning system for new deliveries if new deliveries since last testbot run get list of developers who delivered foreach developer who delivered send 'test started' email checkout code based on last delivery execute build if build problems foreach team member send 'build problems' email redo loop execute smoke test if smoke test problems foreach team member send 'smoke test problems' email redo loop foreach developer who delivered send 'test successful' email redo loop
A small disadvantage of the testbot I've just described is that it heavily relies on e-mail notification. Oftentimes, developers complain about the flooding of e-mails that the testbots send, just because some developer caused a Lint warning on a target they are only marginally interested in.
Thus, I crafted an extended version (see Listing Two) that differentiates between major and minor issues:
loop forever sleep for 5 minutes check versioning system for new deliveries if new deliveries since last testbot run get list of developers who delivered drop 'build in progress' cookie (target.current) foreach developer who delivered send 'test started' email checkout code based on last delivery execute build if build major problems drop 'build major problems' cookie (target.lkb) foreach team member send 'build problems' email redo loop if build minor problems drop 'build minor problems' cookie (target.lkb) foreach team member, architect, project lead send 'build minor problems' email redo loop execute smoke test if smoke test problems drop 'smoke test problems' cookie (target.lkb) foreach team member send 'smoke test problems' email redo loop drop 'test successful' cookie (target.lkgb) remove 'build in progress' cookie (target.current) foreach developer who delivered send 'test successful' email redo loop
In addition to the e-mail-notification mechanism, there is a client program called "Observer" (implemented in Perl/Tk) running on all project member's workstations. The Observer monitors the state of all targets. The following information can be derived from Figure 3:
To provide Observer with build and test state information, the testbots drop cookies on a shared file server directory. Figure 4 presents the contents of the cookie files for a particular target. The Observer regularly checks this shared directory for changes and displays the information in a user-friendly dialog.
By employing this mixed push/pull (e-mail/Observer) scheme, we significantly reduced the number of e-mail broadcasts, thus improving developer happiness. Moreover, the Observer is a nice tool for project leads because they get constant feedback on the quality of their and other leads' projects.
Once you have build and test automation, as well as a testbot framework in place, you can assign all kinds of useful tasks to your testbots. Why not have a testbot execute all of your system tests, stress tests, and performance measurements many more times than you used to? On our project, we have a testbot that takes care of executing all of the thousands of developer (unit) tests. It iterates over a list to which developers add references to their developer test suites. Of course, these extended tests take days to execute, implying that you cannot test individual check-ins but rather whole sets of check-ins. Still, if something goes wrong, you know exactly who made changes to the code base and who is likely to be the culprit.
Delivering functionality in small increments is the preferred way of developing systems these days. Especially on medium to large projects, testbots help ensure code quality and reduce project risk through frequent testingmuch more testing than any human being is able to bear. Equip your testbot with a friendly UI and it will be, as the Sirius Cybernetics Corporation would put it, "your plastic pal who's fun to be with."
DDJ
CruiseControl
Shortly after I had finished this article, the open-source project CruiseControl (http://cruisecontrol.sourceforge. net/) was brought to my attention. So far, I haven't had a chance to use it in a project, but from what I've seen on their web site, it looks like a solid tool to me. It comes with e-mail notification support and reporting facilities. Before you reinvent the (build)loop, you might want to give CruiseControl a try. Let me know what you think of it.
R.H. |
loop forever sleep for 5 minutes check versioning system for new deliveries if new deliveries since last testbot run get list of developers who delivered foreach developer who delivered send 'test started' email checkout code based on last delivery execute build if build problems foreach team member send 'build problems' email redo loop execute smoke test if smoke test problems foreach team member send 'smoke test problems' email redo loop foreach developer who delivered send 'test successful' email redo loop
loop forever sleep for 5 minutes check versioning system for new deliveries if new deliveries since last testbot run get list of developers who delivered drop 'build in progress' cookie (target.current) foreach developer who delivered send 'test started' email checkout code based on last delivery execute build if build major problems drop 'build major problems' cookie (target.lkb) foreach team member send 'build problems' email redo loop if build minor problems drop 'build minor problems' cookie (target.lkb) foreach team member, architect, project lead send 'build minor problems' email redo loop execute smoke test if smoke test problems drop 'smoke test problems' cookie (target.lkb) foreach team member send 'smoke test problems' email redo loop drop 'test successful' cookie (target.lkgb) remove 'build in progress' cookie (target.current) foreach developer who delivered send 'test successful' email redo loop
Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.