Why Did It Work So Well For Us?
This system worked well for us. Our small, tightly focused group quickly saw the benefit of the system, embraced it, and made it more useful. Without that, no matter how technically good the system was, it would have failed.
We internalized the idea of the test system. Rather than seeing it as something that happens after software is written, it has become part of our software design and development process. When we first discuss features or components of our system, we discuss how it will be tested and how it will be folded into our test system. Our software requirement documents include comments on how features can be tested properly and completely in an automated fashion. If something is likely to be too difficult to test, it gets redesignedtesting has become that important to our process.
We conservatively estimate that the test system saves us three man-days every workday. Consequently, we do not see maintenance of it as a chore or an expenditure. We immediately see the benefit from what we've done and are immediately rewarded. In our view, time and money are saved. That was valuable in getting the project started. We don't have a large IT or testing staff to develop the system, so we have to effectively steal time and energy away from other (sometimes critical) projects. This was made easier by making the payoff immediate rather than needing to make the argument that an investment was being made and payoff would come eventually.
Paradoxically, if you can show engineers how you can save them work and then actually deliver, they are willing to put great effort into projects like this. This let the project snowball until it quickly became a critical component in how we develop software and plan engineering activities. All this was realized in hindsight, of course. At the time, we didn't know that this would pay off in such a way or that it would be so well received. It was initially an experiment.
Without the automated test system running all the time, our programmers would not have had the time to perform some of these "by-hand" tests. Problems with usability and documentation were discovered that wouldn't have been found otherwise. The end result is that our system improved all aspects of our testing and developers were able to spend time on other tasks.
Editor's Note
In 2007, FSMLabs sold RTLinux and RTCore to WindRiver, but licensed back rights for use in the enterprise field. FSMLabs continues to use and improve Sparky.