Hardware and Technology for Testing
Again, our software is based on the operating system RTCore, which runs the bare hardware and lets other nonreal-time operating systems (what we call "general-purpose operating systems" or GPOS) run when no real-time task needs to run. This way, our software is an operating system itself but runs cooperatively with other operating systemscurrently Linux, NetBSD, and FreeBSD.
We needed to test RTCore when interacting not only with each of these operating systems but when running on Intel x86, four distinct classes of PowerPC, StrongARM, XScale, ARM7, ARM9, Alpha, and MIPS. These processors range from four-processor VME-bus-based systems used for industrial and military applications down to MMU-less ARM7 processors that run for days on battery when tracking the movements of mountain lions in the Florida Everglades.
We have to push each piece of hardware to the limits of what it can be expected to provide, but not beyond. Spurious failures due to stressing the system too much would lead either to far too much intervention by us to address the problem, or to complacency by neglecting an error when the test system reported it. False errors are simply unacceptable because they would seriously damage the trust we have in the test system.
This required that we compile a detailed list of our expectations for each piece of hardware, then test against that. That's no small task, but is extremely important. When starting on a project like this, care should be taken to ensure that this is done the first time, and continued, with all the diligence necessary.
The test infrastructure hardware itself was made up of standard parts available from the Internet. At the heart of our system are X10 power control modules. These devices connect to a host computer via RS232, turning "receiver" X10 modules on/off. Normally, these are used to turn appliances on/off but we used them to control computers.
To log events in our system, we use serial (RS232) consoles for everything. That means we have about 100+ serial lines that we have to manage and log data on. We started off with inexpensive USB 8-way RS232 devices, but eventually switched to Cyclades 32-way serial devices.
Tracking tests, bug fixes, and changes to the system with something this large and complex isn't easy. If your revision-control software isn't up to the task, then it's even harder. We use BitKeeper to manage our source code, test results, and other data. It's an easy-to-use tool that lets us script things beautifully.