Specifications form the foundation of engineering: If you don't know exactly what you're supposed to build, it's hard to know when you're finished. Worse, you'll probably wind up building The Next Wrong Thing in the most expensive manner imaginable.
The level of detail required in a specification depends on the type of project. A small, homebrew embedded system that monitors rainfall and controls your lawn sprinklers (don't laugh, I've seen 'em) might require only the proverbial back-of-the-envelope sketch and perhaps a page of pseudocode. You'll need much, much more than that to cobble up the engine control system for a new car.
A trip through the aisles at this spring's LinuxWorld Expo (LWE) revealed three interesting items related entirely to embedded specs:
- First, Linux has become the unquestioned 800-kg embedded penguin, regardless of its original design. It may not be particularly graceful on land, but it looms alarmingly over the horizon when seen from anywhere within embedded space.
- Second, you can describe anything as "embedded" if you get your marketeers up early enough in the morning, fortify them with stout buzzwords, and lock them in a meeting room until they've finished the product spec sheet.
- Third, nobody can define what "real time" means, other than that it identifies what their product does in comparison with other contenders.
So, let's have a little talk, starting at the top.
The Scope of Specs
In the beginning, embedded systems ran pretty much on custom hardware, if only because that's pretty much the only kind of hardware there was. I've read arguments that IBM's mid-50s SAGE, the continent-spanning, early-warning radar and interceptor control system that guarded the transpolar approaches to the U.S., was the first large-scale embedded system. While I'm not convinced, it makes a good tale.
About two decades ago, when mass-produced, single-chip microcontrollers finally made it possible to put a computer in every teapot, heavy-duty embedded systems were still built from scratch. The Space Shuttle's flight-control computers are, for example, triply redundant with a completely independent fourth backup. In fact, NASA's specs required both hardware and firmware diversity to rule out a common-mode failure: If three identical modules come up with the same wrong answer, redundancy doesn't buy you anything. Avionics cannot use off-the-shelf solutions, for well and good reason.
Back then, midrange embedded systems were neatly wedged between the proverbial rock and hard place, where standard hardware wasn't rugged enough, full-up custom hardware was too expensive, and the problem specification pretty much required a multitasking OS. There was tremendous pressure to evolve embedded operating systems that could run in severely limited RAM on rather pokey CPUs, while still delivering reasonable performance.
So you saw a fairly neat division of embedded systems. On the low end, one good engineer could pound out microcontroller hardware and firmware in a few months, tops. In the midrange, semicustom hardware plus a semistock OS plus your application plus a small team of engineers and programmers made sense. On the high end, as always, anything went.
What's new and different is that, about two years ago, Moore's Law drove the cost of stock PC hardware around the knee of the price-performance curve. Suddenly, mid-range solutions became feasible on teapot-sized hardware, while simultaneously, the specifications for firmware expanded to require Internet connectivity for every teapot on the plant.
Some things, however, didn't change. Engineers are only about as smart as they've ever been, managers still have trouble keeping specs stable long enough for programmers to hit the target, and one person can only produce so much good output per day, even when working on (mercifully discredited) Internet Time.
In short, the job's now a lot bigger. To produce more output in the same time, you must begin with larger building blocks.
Enter Linux.
Money's too Tight to Mention
There are, I think, two main reasons for the sudden popularity of Linux in embedded systems — money and money. Of the two, money is probably the more important.
Money, in the form of unit cost, drives many decisions in the embedded space that otherwise make no sense at all. With a large-enough production volume, the cost of parts and assembly labor makes up nearly all of the per-unit cost, and development costs amortize out to nearly zero. When the volume is large enough, you can afford almost anything that keeps the cost down.
Once upon a time, I worked with a guy from the RCA television division. He said their engineers were periodically ordered to reduce the unit cost of a TV chassis. They'd gather in a meeting until they agreed how to take 25 cents out of the schematic without changing the overall specifications or performance. Those meetings could get ugly, indeed.
With embedded systems now at the core of nearly all consumer widgetry, a few dollars of additional price can doom a product against its competition. If you don't think so, ask any shopping bot.
Money, in the form of intellectual property licenses and royalties, appears directly in each unit's cost. Obviously, if you want to sell a product for $99.99 at Best Buy, the IP in each unit can't cost twenty bucks. Ideally, you want the IP cost amortized across the whole production run.
This collides squarely with product specs demanding features your team can't produce. The skill set required to write embedded teapot code doesn't necessarily extend to writing and debugging a TCP/IP stack. You're probably better off buying the stack and getting on with the teapot.
But then you pay an IP royalty on each teapot. Oops.
From the buzz I saw at LWE, embedded Linux offers large building blocks at low, perhaps zero, cost. Can you simply add a line that reads "Linux Therein" to your product specs and get results?
Let's check a data point.
A Penguin Embed?
ZFLinux (http://www.zflinux.com/) showed off its Mach-Z at LWE: a 128-MHz, 486-compatible, ISA-bus system on a single chip. It's fast enough to handle Linux, compatible enough to run stock PC software, and affordable enough to put a real PC in your teapot.
For rather large values of teapot, that is. One Mach-Z chip costs about $50.00 in moderate quantities and $33.00 in production lots. The price includes a royalty for the on-chip, PC-compatible BIOS that sets up the hardware and provides all the usual system services.
In addition to the Mach-Z chip, you'll need external DRAM, a NIC or modem, the circuit board, and a few other odds and ends. Let's assume, for the sake of argument, that you could put together a workable embedded Linux system for $75.00 in large quantities.
You must also add the cost of the teapot hardware that makes your product unique, a power supply, and still more odds and ends. Suppose the total unit cost winds up at $125.00.
After you multiply by enough markup to give everybody in the distribution chain some profit on your product, you have a rather expensive teapot. Now, in point of fact, there are markets where a half-kilobuck samovar actually makes sense, but it probably won't be a big hit at Best Buy.
The folks at ZFLinux have certainly done their homework, so we may safely assume the Mach-Z price lies near what you'd pay to assemble the same function from smaller pieces. Remember, however, why we don't want to start with smaller building blocks?
In short, to take advantage of Linux you need a decently peppy CPU, lots of memory, and other accouterments that drive up the hardware cost. Even if the software is free, what it runs on isn't.
This brings up a paradox: I think Linux will compete most successfully with other, not-so-free, operating systems in projects where its vaunted per-unit royalty advantage is moot: gear for markets where final price is not the overriding issue.
So, even though the number of nominally "embedded" Linux products at LWE was a wonder to behold, after I extended my gaze up to the scale of good old SAGE, it all made sense.
Real-Time Specs
Given that Linux is most appropriate for mid- to upper-range embedded projects, what about real-time applications? I saw three orders of magnitude between the "real time" specs from various vendors at LWE, so there's obviously a difference of opinion.
The Linux kernel continues the UNIX lineage of minicomputer operating systems. The scheduler within the kernel allocates CPU time among many different tasks, ensuring that each receives a fair share of the resources and preventing any single task from hogging the box.
That optimization is diametrically opposed to the needs of real-time scheduling, where a specific task must gain control right now and run until it produces the appropriate output. Fairness isn't part of real-time scheduling — getting the job done is.
I once wrote some 8051-class firmware for a project that collected data from a manufacturing line. The specs called for absolutely no data loss: We implemented the classic four-phase handshake nearly everywhere. Data moved through a multidrop RS-485 network to a host PC where it was timestamped and recorded to disk.
That PC ran the then-new Windows 95. I didn't think Win95 was up to the task of collecting data and storing to disk while never, ever missing an incoming serial interrupt. They set up a test system, wrote some code to output a parallel-port blip for every received byte, and hosed it down with serial interrupts.
Sure enough, the PC took interrupt handling vacations that ranged up to hundreds of milliseconds. The project imploded for other reasons, but I still recall their surprise at those gaps in the scope trace.
The folks at MontaVista (http://www.mvista.com/) wrote an interrupt latency measurement tool that shows stock Linux, hot off the CD on a low-end x86, has a latency of tens-to-hundreds of microseconds. Pretty good, eh? The problem isn't so much the average latency, but the occasional peak, which can reach hundreds of milliseconds under perverse conditions. If you must catch every interrupt and your interrupts arrive more than once in a while, this simply won't suffice.
In fact, periodic interrupts pose a particular problem. Even with acceptable latency for a single interrupt, the total path length through the handler, kernel, and your code may be so long that you miss the next interrupt. Always ask about the maximum sustained rate and remember to add the effect of your code.
There seem to be four ways to tighten things up: Patch the kernel and scheduler, replace or amend the scheduler, jack up Linux and slide a hard real-time OS underneath it, or (my favorite) do the hard stuff in hardware.
I want to look into those approaches more closely. For now, remember that the key specification for a real real-time system isn't the average (or typical) case, but the worst case. That can be a hard number to come by, for reasons both good and bad.
Reentry Checklist
If you're a professor teaching an embedded systems course, contact Lineo's Helen Chang ([email protected]) to learn about their Academix support program. It's a hardware-software-courseware package that helps students understand why and how digital bits interface with the outside world. If that interaction seems obvious to you now, think back to the time when it wasn't!
I'm not beating on the Mach-Z, by the way. It's a neat chip and I think they'll be successful in projects where its unit cost is appropriate. The ZFLinux web site has more info, including some hardware reference designs and manuals.
ZFLinux also introduced a spiffy term for the Mach-Z's automatic firmware reloading: Fail-safe Boot. If (when!) malicious forces corrupt your flash ROMed firmware, an on-chip routine reads your emergency recovery code from a bombproof EEPROM. Your code then phones home, downloads a fresh slug of firmware, flashes the ROM, and restarts normally.
Back in 1962, long before "embedded systems" entered the lexicon, Burdick and Wheeler wrote Fail-Safe (ISBN 088001654X). The novel details the horrific series of events following a malfunction in the hair trigger system that controlled our nuclear bombers. Any embedded developer who can read it and not get cold sweats probably shouldn't be an embedded developer. When you're finished, ask yourself if any contemporary President could do what obviously had to be done.
DDJ
