As a rule of thumb, any operation requiring simple logic, rapid reflexes, and tireless attention is a perfect candidate for embedded control. We humans aren't good at any of those, notably the tireless attention part, so it makes sense for a machine to help us out.

There is, however, a countervailing force, governed by the Law of Conservation of Perversity, that ensures that we can't make something simpler without making something else more complex. Stated more succinctly, whenever we push on the universe, it pushes back.

Broken Free

Take, for example, the Antilock Brake System (ABS) found in most new cars. Given a rotational sensor at each wheel, control over the brake cylinders, and a liberal dollop of embedded logic, ABS can extract the maximum stopping force available from the pavement beneath it. That's the promise, anyway.

The reality is a bit more complex. Here in the Hudson Valley of New York, winter travel crosses the occasional patch of ice or snow. ABS works fine on dry pavements, which we enjoy most of the year. On ice, however, where you simply don't have much traction to begin with, braking becomes a delicate proposition.

Stopping on ice requires gentle pedal pressure that rides the bare edge of a skid. If your car begins skidding, releasing the brakes doesn't help because the tires won't restick to the slick surface passing underneath. After your car comes unstuck, Isaac Newton becomes your chauffeur.

Ice braking is a learned skill, one I spent days acquiring by sliding our family Studebaker around on the high school parking lot (to the horror of my parents, with whom I did not first consult) and have probably lost by lack of practice in the intervening years. Stopping requires a careful touch on the brakes, steady nerves, and plenty of room. One early discovery: You can't stop abruptly, no matter how good you (think you) are.

This past year, much to the amusement of my friends, I equipped our new van with four top-rated winter tires. At the first opportunity, I tried out their alleged excellent ice traction along our driveway's somewhat icy 400-foot length.

As I eased down on the pedal, the van slowed pretty much as I expected. However, when the ABS noted a slipping tire, it abruptly took control, released that brake, then abruptly applied it again — which knocked the other tires loose from the ice. Fortunately, nobody was injured while laughing at my exploits.

Repeated trials produced the same sequence of events. I could not stop the van as I expected; the ABS logic simply wouldn't believe I knew what I was doing or that the van was slowing down as I intended. Maybe I didn't, maybe it wasn't, but there was no way to convince the ABS to give me the final say in the matter.

Worse, I think, is how ABS trains you to just stomp on the brake pedal and let the automation sort things out. We no longer have the opportunity to learn how pedal pressure affects stopping distance and can't develop a feel for braking under adverse conditions, let alone normal driving.

Does that make ABS a bad thing? Not at all. In the vast majority of situations, the logic embedded in the ABS controller is a net win: On dry pavements, on wet pavements, and in most day-to-day driving, I think it does a better job than I would. But there remain a few situations where I'd like finer control than it will give me.

That's the Law of Conservation of Perversity at work.

Performed an Invalid Operation

The Law also applies to desktop PCs, where most users have no experience with a system that doesn't sport a GUI: DOS is well and truly dead. Only the odd geek knows or cares about the command-line underpinnings of the OS. If it's not simple, it simply won't sell!

And in truth, using Windows (or, yes, a Mac) for nearly all day-to-day operations is much simpler than doing things the bad old way. Most people, myself included, generally use PCs as tools rather than learning experiences; a point-and-grunt interface serves wonderfully well for that purpose.

But the Law of Conservation of Perversity tells us that, as we spend more time dealing with easy-to-use GUIs, we become correspondingly less familiar with the underlying machinery. When anything goes wrong that's not handled by a GUI tool, we're sunk without a trace. Simplicity and ease of use do not necessarily imply ease of maintenance. Indeed, the opposite may be truer.

As with my mother, who will be 81 this spring. She does fine with e-mail and web browsing, but has little patience for the fine art of system administration. Imagine what transpired when we attempted to switch her to a new ISP and the installation program crashed, killing off her previous ISP account without setting up a new one.

The diagnostics weren't particularly helpful, either. Honestly, who among you believes that viewing a stack dump of the offending program assists in finding the problem? The installation program supposedly produced a few desktop icons that simply didn't show up anywhere. Attempting to rerun the program didn't work either, as the shattered remains of its previous execution looked successful enough to prevent it from running again.

Bear in mind that Mom's circle of friends doesn't include anyone who's up to speed on Windows repair work and that I'm six hours away by Interstate. It was not a pretty scene, although a remote control program and several hours of cheap long-distance rates came in handy. Things worked out well enough in the end, with plenty of learning experiences for all.

Now, would insisting that Mom learn to handle all the possible failures before she begins using e-mail, browsing the Web, or changing her system have avoided the problem? Should we expect her to learn all those tricks it's taken me years to master? Nope, she's an appliance user and that's what she wants from a PC.

You cannot run the Law in reverse by insisting that people learn all the details before trying anything new. All the folks who think that's a really good idea are already geeks; the remaining populace has better things to do than memorize the intricacies of hammering out PPP connections from scratch.

(And by the way, the Internet Connection Wizard must have been on strike. That particular effort at simplifying things did not help in the least.)

Simplification also helps explain why Linux has such an uphill row to hoe on the desktop. Both KDE and Gnome, pretty and useful as they may be, lacquer only the thinnest veneer over a relentlessly command-line-driven OS. Without a deep knowledge of the underlying and inherently complex system structure, you can't get very far.

But GUIs and ease of use remain good ideas, even if the Law tells us what must happen. Some things, notably UNIX, just can't be made simple. There's an obvious market opportunity here for someone with a burning desire to spend a lifetime in a hands-on craft business, helping set things right after they go wrong.

No Fury Like That of an Unjustified Assumption

Most major (and many minor) appliances nowadays sport deeply embedded microcontrollers to implement their control logic. By "deeply embedded," I mean that if you don't pop the covers, you'd never know what's inside. I do, of course, but I'm that type of guy — with a complete set of Torx security bits for the tough ones.

Inside our water softener, a microcontroller measures water flow and schedules each regeneration to minimize salt consumption. The softener it replaced, installed in the early '70s by the original owners of our house, performed much the same function, albeit driven by a now-irreplaceable timing motor, a worn plastic gear train, and an array of slightly leaky rubber-covered valves.

Mechanically, the new softener has fewer parts, simpler hardware, and definitely easier assembly at the factory. The microcontroller handles all the measurement and control logic, presents time and status on an LCD panel, and controls a single motor that drives one rotary water valve. What's not to like?

This one probably won't last a quarter-century, if only because the unprotected circuit board lives about six inches from a hopper holding 200 pounds of salt. There's a reason why ship manufacturers mount oceangoing electronics in watertight enclosures with stainless-steel hardware and charge premium prices. Although the softener's electronics compartment is shielded from salt dust, I have my doubts about how effective this can be in the long term.

Using the Law of Conservation of Perversity, you can easily determine why improving manufacturing simplicity might produce a system that's less reliable overall.

Been Terminated

Something must yield when mandatory ease of use collides with relentless cost reduction. I predict that error detection and handling will reveal the dark side of embedded design: When things go wrong, the symptoms and error messages give no clues as to what to do.

Doubt it? It's already happening!

Early in 2000, many people discovered their new VCRs displayed the wrong time. Unlike older VCRs, many had no manual time-setting buttons because their microcontrollers decoded the time from a digital bitstream tucked into the analog video signal's vertical blanking interval. Obviously, if it's digital, it must be correct, right?

Depending on which cable system the VCR owners subscribed to, their clocks could be slow by one, two, or three hours. While that may be easy to understand, other VCRs were off, either slow or fast, by a few minutes, exactly 24 minutes, or amounts that varied from day to day. That's not to mention their new TVs, which might be off in a different direction by a different amount.

Most owners, it seems, had no idea what to do and simply put up with incorrect times, much as they had with blinking 12:00 digits. Technical owners (you can almost hear "I can fix anything" in the background) called their local stations, cable companies, and national networks with much the same results: either complete disbelief or an equally complete inability to find the cause.

Eventually, the truth(s) came out. PBS stations (are supposed to) encode the correct time of day in their local signals and the VCRs (are supposed to) scan the incoming channels to find a valid signal. Local stations (are supposed to) override network time codes to maintain the correct time-zone setting.

From that brief description, how many points of failure can you identify? Half a dozen? That's about the size of it; each and every possible failure happened at least once, occasionally simultaneously.

My favorite: The guy who knew how to set the timecode generator at the local PBS affiliate station didn't work there any more. By the time the clock oscillator drifted 24 minutes away from real time, nobody even knew what the timecode generator looked like, let alone how to set the time.

Which, by the way, required jacking a laptop into the generator's serial port and following the telephoned instructions of someone who knew how to drive its command-line interface. Does this sound familiar?

And we think we can make an ad hoc RF network of embedded widgets not only work reliably, but deliver meaningful diagnostics when they fail? Meaningful to the nontechnical users who'll buy these things based on a promise that their lives will be simplified?

It could happen...

Reentry Checklist

Unlike the embedded world in general, aerospace folks study their failures, report on their mistakes, and correct the causes. As the overall airline safety record shows, they're generally doing it right and the failures are few and far between.

After decades of hard work, they're well past the "silver bullet" stage that embedded systems design has yet to approach. We embedded systems designers must do a better job of error recording and presentation, with an eye toward making it much easier for ordinary folks to figure out what's wrong and fix it the first time.

For motivation, search for +"cockpit voice recorder" +transcript. You'll find well-trained folks doing their best to keep a serious problem from recurring, right down to the bitter end. And we begrudge a few KB for logging?

NASA web sites (start at the obvious http://www.nasa.gov/) include their own postmortems on the Mars and lunar missions, NEAR, and others. Would that they had sufficient resources to make penny-pinching a thing of the past.

The IEEE Spectrum magazine's articles (October and December 2000) on the VCR autoclock problem shows why simple solutions often produce complex problems. Imagine what a complex solution could do!

DDJ