Testing One Two

By that definition, I don't know anyone who isn't a bungler. Nothing with any complexity at all works perfectly the first time, precious few work the second time, and, as a good rule of thumb, anything involving software takes even longer. Perhaps I travel with the wrong crowd.

Software testing should be, at least in principle, completely unnecessary. We should be able to produce programs that perform their specified functions (and only those functions!) with predictable run-time performance, delivered on time and within budget. The fact that such an outcome occurs only in exceptional situations should tell you just how bad things are.

I heard several talks and saw products devoted to testing at last year's Embedded Systems Conference, Software Development Expo, and Real-Time Linux Workshop in Boston. Before we see what's up, let's take a look at what makes testing embedded systems so difficult.

Full-Up Testing

Back in the days of pure analog circuitry, you could do a decent job of testing with a rack of signal sources and an oscilloscope or two, perhaps deploying a spectrum analyzer for the really tough cases. Access to test points within a hand-wired chassis posed few problems, as long as you didn't electrocute yourself on the bias supply voltages.

If you can find an analog voltage in a contemporary system, you're probably looking at the power supply or an external signal on a connector. The drive to digitize everything means that the voltage on a single pin doesn't tell you anything about what's going on. In fact, without knowing the signaling protocol, even recording a pin's activity may be completely meaningless: An oscilloscope won't tell why a digital audio output sounds funny.

Figure 1 puts another problem in perspective with an oscilloscope probe atop a current-generation IC with 552 (count 'em!) solder balls populating a 1-millimeter grid. The circuit board may have several dozen internal signal layers, so relatively few signals travel across the surface layers. Even the miniature Tek probe described in my October 2002 column can't help much here!

Contemporary chips include a serial data path that pipes internal registers off the chip through four or five pins. A daisy-chain connection through all the logic chips on the board provides relatively low-speed access for hardware testing during chip and board production. This technique, formerly known as JTAG and now as IEEE 1194.1, lies behind the background debug mode (BDM) and on-chip emulation features of many processor chips.

BDM restricts itself to the registers found in the programmer's model of the CPU, not the entire state of the board. Capturing and reading JTAG/BDM data can be either nonintrusive or blunt-force trauma depending on the chip's interconnections, so make sure the hardware folks know what you need.

David Stewart of Embedded Research Solutions (http://www.embedded-zone.com/) presented "The 25 Most Common Mistakes with Real-Time Software Development," among which was his observation that essentially all glitches arise from task deadlocks, memory overwrites, stack overflows, and similar software maladies. Essentially, no glitches arise from hardware problems, particularly after you get the second or third system up.

Embedded software's lowest level deals intimately with hardware and depends on precise signal timings. Bringing up such code on the first machine can reveal problems with the software specification, the incoming signals, the hardware design, or the hardware itself: You cannot assume anything works as it should, even when the documentation says everything is fine.

Stewart highly recommends keeping a glitch notebook. Whenever something interesting occurs, you record what happened and what was going on at the time. My experience says a paper notebook tucked in your pocket works best because quick sketches come in handy, but a PDA or even a text file on an unrelated system might suffice. Creating entries must be trivially easy, however: Having to locate a keyboard makes for a sparse log.

After the first system works, it becomes remarkably easy to believe that those intermittent glitches can't possibly indicate a software error: We've looked everywhere in our code and considered every possibility. You know the feeling, yes?

My esteemed wife offers an example of how wrong such knowledge can be. Some years ago, she was involved with a mainframe operating-system release that experienced a catastrophic crash every now and again. It didn't crash often, it didn't crash regularly, but it did crash. All the usual proactive debugging efforts to find the problem failed, so one guy started analyzing the accumulated core dumps.

After nearly five months of all-day, every-day scrutiny, he located a status flag update. Once in a great while, that instruction's address pointer held an incorrect value, so, somewhat less often, a single bit changed in a randomly chosen word within the operating system's data structures.

Verily it is written that in kernel space, nobody can hear you scream...

She observes that this problem didn't make it out the door to the customers. You may insert the obligatory desktop PC operating system slur at this point.

Avoiding such a debacle requires both solid design and robust error-handling routines, although I'm not sure how you'd go about identifying a random bit flip. Embedded systems face the additional restriction that they might not have a place to store those core dumps and might not be able to send them anywhere for analysis.

You must verify those error handlers early in the project's implementation because they'll probably get a lot of use, then leave them running in the deployed system. Several speakers pointed out that if your project can't withstand the additional overhead of exception-handling logic, you're ignoring a much more serious problem.

The expert consensus on error handling boiled down to two simple facts. First, if you can't handle an exception, don't bother testing for it. Second, if you can't handle it, you're in trouble because code in deeply embedded systems can and will fail for reasons that have nothing to do with how good a job you did simulating the system before release.

Chuck Allison (http://www.freshsource .com/) observed that nobody, but nobody, checks the return code from printf(). Indeed, quite a few folks in the audience didn't know what that return code was. Pop Quiz: Do you? Deep within an embedded system, it's entirely possible that a driver might indicate its output device has failed. What would your code do?

Integrated Testing

A good project starts with a good design, but the final result must match up with those specs. The Stride integration and testing framework from S2 Technologies (http://www.s2tec.com/) attempts to solve the problem of making sure you actually did what you set out to do.

Stride insinuates itself between all your modules and into the external APIs of your system. Normal debuggers can handle instruction-level situations while Stride shows you the global operations and data flows. While it requires special source code and a separate compilation to ooze into the interfaces, it's said to be independent of your hardware, RTOS, and development environment.

In principle, you should be writing the doc and the test cases first, so that you can verify each module and subsystem as you actually write the code. Assuming this is the case, Stride lets you apply those testcases at a fairly high level, even working from Matlab models of your system.

You can simulate missing components using scripts, run modules on development boxes or the target hardware, and verify that everything is producing and consuming the appropriate data. Although it's not mentioned in the writeups I've seen, you most likely can't deploy the Stride code in the finished system.

Stride is obviously intended for large projects with complex interfaces rather than for smaller systems. On the other hand, they point out that mobile phone development lies in the mega-dollar and mega-LOC region, so big problems come in small packages these days.

Presenters at both the ESC and Real-Time Linux shows observed that even systems with protected memory and excellent module-to-module isolation can exhibit tight temporal coupling, as each chunk of code monopolizes the CPU at the expense of the others. You may find that your code simulates and tests perfectly, while still failing every now and again in the field.

For example, OS tracing code salvaged the 1997 Mars Pathfinder mission. The only external symptom, a watchdog timer reset after an intermittent system lockup, didn't provide much evidence to analyze. An identical ground-based system run with tracing enabled (always ship the debugging code!) in the VxWorks OS eventually pinned down the problem—a sporadic priority inversion on a system mutex.

Now imagine a debugging session with a 40-minute delay between each command and its results, which is what confronted the Pathfinder team. That duplicate spacecraft on the ground was the only way to actually put the debugging information to use. Talk about a high-latency channel.

It's worth noting that the few watchdog resets observed during Pathfinder's exhaustive final tests were discounted as being due to, well, hardware problems. Sound familiar?

Although it's straightforward, if not easy, to verify static mainline code, verifying its dynamic behavior and the subsequent error handling depends on your ability to foresee timing issues, contention for resources, and outright hardware failures. There are no easy answers for any of this, so vigilance may be the best we can hope for.

Executable Specs

What if you could go directly from specifications to executable code? That's been the Holy Grail of software development since about the time Grace Hopper's Cobol development group hammered out the first "English language" computer interface. Recall that Cobol's original target users were business managers, not programmers, and you'll have some notion of the difficulty they faced.

Matlab, from The MathWorks (http://www.mathworks.com/), started out long ago as a mainframe Fortran (!) front-end to the standard matrix-math libraries of the day. Since then, it's become the Swiss Army Knife of workstation mathematics, accreted simulation, state machine and logic tools, and has now extended its reach into programming.

What's of interest to embedded systems folks is the Real-Time Workshop Embedded Coder, which can convert a complete control system model into C code. The resulting code can drop directly into your system or, with a bit more effort, can interface with the OS of your choice.

While this isn't exactly hands-free coding, it does prevent flipped signs and missing conversion factors along the way. Even better, going directly from model to code provides an absolute link between the high-level design and the low-level bits; once you've verified the design, the code must be correct.

Well, correct within the limits we've discussed; a perfect model may still founder in the real world. Even so, not having to deal with code testing might make the whole exercise worthwhile.

As nearly as I can tell, this works best for math-heavy control systems, the sort Matlab was designed to model. The level of effort required to describe state machines, nonmathematical logic structures, and error handling seems to be roughly equivalent to building the code directly. It is definitely not a "do what I mean" system.

That may be missing the point. For a certain class of problem, you can state your requirements, build and verify a system model, then go directly to known-good code that plunks into your gizmo. This might very well be a big enough win to justify the effort of writing your code in Matlab's style rather than your own.

Hearing Clearly

Judging from the notes I've received after the January 2003 column on high sound levels, many of you take hearing protection seriously.

The reason you can't hear very well for a while after a concert has nothing to do with those little muscles in your ear, although they do cause that cringing sensation. The hair cells that convert cochlear fluid motion into nerve impulses sport even tinier structures called "stereocilia" that act as mechanical amplifiers. Loud sounds dishevel the stereocilia and reduce their efficiency until they comb themselves out again. The before-and-after pictures at http://www.musiced.org.uk/features/ hearing/ will definitely induce you to wear earplugs. Thanks to Robert Masta for setting me straight.

A recent Crutchfield catalog (http://www.crutchfield.com/) asks "How much bass can you take?" and answers with three subwoofer/amp packages. Their numbers don't add up, as the "145 dB" package includes subwoofers with an 87 dB/W sensitivity and a 400 Wrms maximum rating. That's only 113 dB per driver or 116 dB for the pair. The 1 kW amp can slightly overdrive the speakers to reach 117 dB. At a kilobuck for the package, this may be a tax on folks who flunked math.

For comparison, Shuttle Launch Complexes 39A and 39B each have a 900,000 gallon/minute water-spray sound suppression system that reduces the acoustic pressure inside the Shuttle's cargo bay to 142 dB. Trust me on this: That beater in the next lane isn't in the same category.

Contact Release

E.E. Smith wrote the first "space opera" novels: resolute heroes battling nefarious scoundrels along the galactic spaceways. Much of what passes for popular SF these days descends directly from his Skylark of Space and Galactic Patrol series, both of which remain worthwhile even if their science leaves a bit to be desired.

For writeups on the Mars Pathfinder story, see http://www.splendors.com/tidbits/ mars.html and "A Conversation with Glenn Reeves," by Jack J. Woehr (DDJ, November 1999).

You'll find more than you probably want to know about "Amazing Grace" Hopper at http://www.jamesshuggins.com/h/tek1/grace_hopper.htm.

DDJ