First Steps

Ed's an EE, PE, and author in Poughkeepsie, NY. Contact him at [email protected] with "Dr Dobbs" in the subject to avoid spam filters.

It ain't what you don't know that hurts you, it's what you know that ain't so.
—Samuel Clemens or Will Rogers or Albert Einstein or any experienced engineer

This year, she announced that her summer project would be a line follower and that she wanted to learn how to write a "real program" for the robot. I set up the Mindstorms IDE on my token Windows laptop, we cleared off a table in the basement, and away she went, building and programming.

I restrained myself, and after a few days, found her dispiritedly sticking some Lego bricks on something resembling a house. As I had expected, she found learning to program, even using the Mindstorms drag-and-drop graphic IDE, rather more difficult than she had expected. But you knew that already, right?

There's more to the problem than meets the eye and it applies directly to embedded-systems designers.

The Real World Strikes Back

The 12th annual Trinity Firefighting Home Robots Contest went off without a hitch in April, despite (or perhaps due to) a considerably lower profile security presence than last year. I served as a judge for the Junior Division entries, which is open to students through 8th grade and uses the Standard Arena Layout.

That arena features walls covered with white paint or light wallpaper, a matte-black floor with white tape marking the room entrances, and a single candle atop a white disk in a corner of one of the four rooms. The arenas used by older and presumably more experienced contestants have similar layouts with additional physical and optical challenges.

Only two of the 13 Junior robots completed their missions by finding and extinguishing the flame in any of three separate attempts. The remainder suffered various failures, ranging from motionless stalls to autonomous disassembly as protruding wheels or sensors snagged on doorways and corners. While enthusiasm ran high, something was obviously wrong.

Lego Mindstorms sets provide the raw material for many Junior robots because snap-together parts eliminate the need for advanced mechanical skills and a GUI IDE eliminates programming syntax errors. The resulting robots seem to be reasonably functional, at least when they're run at home or school.

Figure 1 shows a typical Lego robot. A rolling ball forms a tripod with the two driving wheels that make steering a matter of motor speed balance. The gray and blue blocks below the fan are a Lego Light Sensor brick, containing a red LED and phototransistor, to detect the white tape demarcating each doorway. An aluminum-foil cylinder above the fan motor shields another Light Sensor that detects the candle flame.

The Lego Mindstorms controller has three motor-drive outputs and three analog inputs, which barely suffice for even a simple robot. Two drive motors, one fan motor, and two sensors leave only one free analog input, to which some students attach a Light Sensor or switch for wall following. Sliding along a wall incurs a penalty, but a robot can occasionally touch the wall for a sanity check or use a non-contact optical sensor.

The contest rules define the arena layout, so, unlike a maze-solving competition, robots can use an internal model for navigation. Multiplying time by speed should produce the distance traveled, although any errors will accumulate as the robots travel through the hallways and rooms.

Unfortunately, Lego motors spin at rates depending on (at least) the mechanical load, battery voltage, and wheel traction. The controller can produce only eight motor speeds and I've found that the fastest four speeds are not only too fast for large direct-drive wheels, but essentially equivalent. As a result, dependable open-loop dead-reckoning navigation poses a major challenge.

The garish colors and pink tinge in Figure 1 come from the gym's high-pressure sodium-arc lamps, which produce a reasonably good illumination with high energy efficiency. However, a spectrometer reveals that most of the visible energy is orange or green, with a slight blue component.

The lamps also produce a tremendous amount of invisible infrared energy, with an emission peak almost exactly at the greatest sensitivity of ordinary silicon photodiodes. In other words, an HP-Na lamp would make a perfect IR illuminator if you filtered out the visible light. The arc producing the light also flickers at 120 Hz.

The simple intensity-measurement circuitry in the Lego Light Sensors weren't designed for the background illumination of the Trinity arenas. The AC modulation produces "distance" readings that vary randomly because the IR background swamps the visible LED lighting up the target.

The hardest parts of the contest involve invisible effects: small motor speed variations and sensor interference. Designing and debugging a robot in a relatively benign home or school room get students past the obvious problems, but the Trinity arenas introduce a whole new level of complexity. Having kibitzed with many teams over the years, I can assert that the symptoms seem inexplicable without an engineering background.

The students building Junior-class robots have plenty of enthusiasm and a good mastery of the basic skills, lacking only the deeper knowledge that comes from experience. It turns out that they're not alone.

That Gurgling Sound

I fished a note from Ventoro ("the market leading offshore outsourcing and offshoring research and consulting firm") out of my Spam folder and read its 115-page Offshore 2005 Research report, which presents the results of a survey of 5231 executives "who would generally be considered buyers of offshore outsourcing services." The results give some idea of what's going on in that layer of management that assumes you can be (convinced to be) cheerful while training your replacement.

Although the main motivation for moving jobs offshore always seems to be reduced cost, the reported results indicate an average saving of about 10 percent, with perhaps 1/4 of the projects suffering a cost increase, 1/5 seeing no change, and the bulk coming in under 20 percent.

In August, I mentioned that software development is on the verge of becoming predictable, with the inevitable result that predictable projects will move to any location producing the same quality at lower cost. It turns out that 50 percent of the survey respondents sought offshore vendors to "Improve Quality" for jobs ranging from call centers to engineering.

They note that offshore vendors with one or more of the usual process certifications have a much higher probability of completing a successful project. If the vendor invests equally on both CMM Level 5, for example, and what's called "customer engagement methodology," the success rate jumps from 1/2 to nearly 2/3.

What's in it for you? The report distinguishes between two groups of workers. Unskilled employees, such as those found in call centers, can be "easily retooled" into similar jobs. Skilled employees, which means software developers these days, cannot be retrained into higher-level jobs like architects or analysts: "simply offering a retraining program will not ensure your ace developer will be a sound project manager."

No surprise there, eh?

Reality Checks

Although the Lego Mindstorms language has the compelling advantage of making things move, it lacks a number of familiar features. Deeply embedded systems typically have restricted I/O and there's nothing unusual with not being able to spin up a console, but explaining how things work without text I/O is a tough exercise.

Our young lady has begun working through some Python tutorials. Python is close enough to the robot control language that the concepts should carry over without too many syntactic issues. She's just grasped the notion of variables, knows how to prompt for input, can display output, and is working on looping. Laugh if you will, but there was a time when that stuff was new to you, too.

Figure 2 shows a Trinity robot bearing down on the inferno of a candle, as seen through a filter that passes only infrared light. An older team scratch-built this robot, which sports industrial IR distance sensors for wall-following navigation. The sensors use modulated IR signals that stand out from the clutter and return dependable distance information without fancy programming.

At this level of complexity, a typical Trinity robot uses a standard microcontroller board programmed in an embedded flavor of C. Some use various Basic Stamps, in which a Microchip PIC microcontroller processes a tokenized Basic program. Stepper motors may replace DC motors at the wheels and you may see rotary encoders on the wheel shafts to provide precise feedback. The construction techniques range from disciplined to hairball, but the students show that they understand the problem.

Five of the eight robots in the High-School Entry Division completed their mission, with three posting scores indicating they were functioning more-or-less as intended. Four of the 13 robots in the High-School Standard Division finished, with scores showing that they had major problems along the way.

The Senior Division, where college students and adults enter some truly complex robots, had 22 entries, of which 14 ran successfully. The first-place finisher scored half that of the runner-up, with the remainder encountering various problems.

The two most challenging Divisions—Expert and Walking—had four entrants each, none of which completed their tasks.

As the challenge level increases, so does the opportunity for mistakes. Whether background light overwhelms a sensor, a buffer overflow trashes the firmware, or a faulty loop runs forever makes little difference: The robot fails to run. The difference, perhaps, is that an experienced engineer knows how to go about finding the problem, after which the repair generally seems simple.

That so many problems only appear in the heat of competition is why we have competitions in the first place. Fancy paper designs and elegant logic models never fail, but actual hardware can present some truly baffling bugs. Building a functional device requires experience of the sort you can't get without hands-on contact.

A common misperception has it that India and China are hotbeds of cheap, skilled talent: people willing to work for wages that wouldn't keep you afloat in any U.S. conurbation. The Ventoro report observes such is not the case and that substituting good processes for good people simply doesn't work. Teams and processes built around the best available offshore staff had a 60 percent chance of succeeding, while companies filling a process-defined org chart with cheap and readily available bodies failed nearly 80 percent of the time.

All the many outsourcing projects have begun bidding up wages, particularly for experienced developers. As the report puts it, "To build a team of qualified employees in an offshore outsourcing market, you must...be willing to pay a premium."

It seems that the 1990s enlargement of the much-maligned H1-B program sucked the most experienced employees right out of the offshore marketplace; they're now U.S. residents, either citizens or holding green cards. As a result, "There simply are not enough senior resources with the true experience needed." In this terminology, "senior resource" means "...someone that knows everything that is not explained. They know all the tricks, all of the challenges and the standards that you can only get from years of experience."

That's someone like you, I trust.

Incidentally, what about the much-vaunted continuous development technique of rotating a project around the globe to pack three shifts into every day? As you'd expect, it doesn't work: "The customer greatly increased the cost to develop the product and failed to prove any time to market improvements."

Where Are We?

Despite what the ads claim, nothing about writing code is intuitive. Converting specifications into a correct and functional program requires skills that we're simply not born with. The evidence suggests that even well-trained people have trouble building code that works.

Embedded systems pose an additional challenge—writing code that copes with the real world. A program that correctly implements a given specification can still fail when the hardware operates in unusual circumstances: The Ariane 5 failure is a perfect example. Writing specs for ordinary business and webbish apps seems difficult until you try writing specs for a physical gizmo!

If you're an engineering sort of bear, one with experience making things work, you might well have a marketable skill in this day and age. The problem seems to be that the folks hiring you don't recognize that experience counts for more than buzzword compliance.

While the offshoring trend will continue, I'm starting to see some indications that experience really should count. If you've got that experience, good for you. If not, well, it's long past time to start moving in that direction.

And if you should be in the management realm, factor in the statistic that four out of five projects staffed with inexperienced bodies fail. When you're looking for experience, starting close to home has some benefits that should appear in the risk-versus-reward matrix.

Reentry Checklist

The Official Lego Mindstorms home base is at http://mindstorms.lego.com/. A quick Internet search will turn up many hotbeds of Mindstorms enthusiasts, including http://www.philohome.com/ and http://www.plazaearth.com/usr/gasperi/lego.htm.

The Trinity Contest web site is at http://www.trincoll.edu/events/robot/default.asp. In addition to serving as a judge, I'm also on the Technical Committee, and in 2006, I plan to hold a contest-floor clinic to help students understand and debug their problems. The photos come from previous years and I don't recall how well those robots worked.

The Ventoro offshoring report is at http://ventoro.com/Offshore2005ResearchFindings.pdf. Other than getting it in my e-mail, I have no relation with them.

We're using IDLE to learn Python, which probably came with your Linux distro. If not, fetch IDLE from http://www.python.org/idle/. The LiveWires course at http://www.livewires.org.uk/python/index.html is a straightforward introduction for young folks, although an experienced oldster must be close at hand to surmount the initial learning hump.

DDJ