Ed's an EE, PE, and author in Poughkeepsie, NY. Contact him at [email protected] with "Dr Dobbs" in the subject to avoid spam filters.
An ad from a few years ago featured a techie poring over a big display with a worried look, thinking "If I could find it, I could fix it!" We've all had that experience: A chunk of code does something unexpected, just once, and now it's time to figure out what happened. Unfortunately, it's really hard to locate the offending section when you can't make the failure happen on command.
Embedded systems take this to an extreme: The symptom may be a mechanical glitch that happens only under the most peculiar circumstances. Ever notice how automotive problems vanish on the way to the repair shop? Now, imagine debugging the code behind a failure that only occurs on the road, where the customers don't know or care that there's an embedded system involved.
The National Highway Traffic Safety Administration (NHTSA) keeps track of automotive recalls in a public database. In the last few years, some new terms have joined the lexicon of transmissions, fuel injectors, lamps, and brakes—software, flash memory, computers, and programming errors.
Let's pop the hood and examine a few familiar monsters lurking inside. The indented paragraphs come from the NHTSA database, condensed and lightly edited for readability.
Corner Cases
Embedded code must produce outputs based on a vehicle model fed by very few inputs: some binary switches and maybe a few analog voltages. A corner case occurs when a program's inputs seem to make sense individually, but collectively represent a condition that the programmer didn't expect. Usually, that happens when the model diverges from reality and the inputs stop making sense.
For example, security systems must distinguish friend from foe. This one mislaid the friends.
BMW 535i 1994: The double-lock feature can engage with occupants and the door/ignition key inside the vehicle. The occupants of the vehicle would be unable to exit either from the doors or from the windows of the vehicle. Drive-away protection would prevent the engine from starting. Dealers will replace the general control module with one containing the revised software to permit window opening with the double-lock engaged and key in the ignition.
I'm not sure how complex a speedometer can be, but fixed-point division is always tricky. Alas, to judge from the motorcyclists around here, any speedo reading is largely irrelevant.
Honda CBR1000RR 2004: On certain motorcycles, some digital speedometers may experience a computer program error, causing the speedometer to indicate approximately 25 percent less than the actual vehicle speed. This condition can result in the vehicle being driven at an illegal or unsafe speed, which could increase the risk of a crash. Dealers will install an updated speedometer and associated components.
Even if the code calculates correctly, there may be other considerations. Ford is particularly sensitive to tire failures in Explorers, so perhaps this one popped up during a code review.
Ford Explorer 1999: The subject vehicles (equipped with the 3.27 or 3.55 axle ratio) have a powertrain control module that could allow the vehicle to exceed the design intent top speed. At these extreme speeds, the vehicle could be capable of exceeding the speed rating of the tire. Dealers will reprogram the powertrain control module.
Even pure, stable analog inputs can cause trouble and this one may be a full-scale calibration error. Liquid float sensors are notoriously nonlinear, so a table-lookup bug may also be lurking in here.
Porsche 911 2000: Incorrect programming of the electronic logic unit can cause an incorrect fuel level. The vehicle could run out of fuel causing a loss of power. Dealers will update the software in the instrument cluster so that the fuel level and the range of remaining fuel are determined and displayed correctly.
Here's a classic corner case, with too many things happening at once.
Nissan Frontier 2002: On certain pickup trucks and sport utility vehicles equipped with a supercharger, under certain conditions, such as full throttle in low gear when the engine is not fully warmed up and at cold ambient temperature, the amount of intake air flow through the air flow meter may exceed the maximum preset diagnosis limit value. This will cause the engine control system to go into the failsafe mode. Under this condition, the malfunction indicator lamp will illuminate and engine speed will not exceed 2400 rpm regardless of the throttle position, increasing the risk of a crash. Dealers will reprogram the electronic control module.
Rust Never Sleeps
Designing mil-spec hardware is tough, but designing automotive electronics is harder: It must work in phenomenally hostile conditions and cost almost nothing. Those conditions include Minnesota winters and Florida summers, seasoned with salt spray and a dash of hydrocarbon solvent. Even back in the good old days, things went wrong.
Volkswagen Rabbit 1975: The fender opening through which the antenna cable is routed may not be properly sealed. As a result, splashed water from the tires could enter the passenger compartment through the opening, enter the fuse relay board and cause corrosion of the terminals. Also, because water is a good conductor of electricity, the starter can be activated accidentally; the high flow of electricity may exhaust the battery within a short period of time. Dealer will inspect and, if necessary, seal the radio antenna holes and replace corroded components.
One way to prevent circuit-board corrosion involves applying a conformal coating to seal the electronics.
Toyota Celica 1982: Soldered terminals of the cruise control computer may develop cracks due to improper application of the coating to the printed circuit board. Consequence of defect: Continued use of cruise control could lead to complete separation of soldered terminals and circuit failure; engine speed would instantly race and vehicle could suddenly accelerate, possibly resulting in an accident. Replace cruise control computer, if necessary.
If your code depends on one input bit, the accuracy of that bit takes on special significance.
Dodge Viper 2003: On certain passenger vehicles, the multifunction switch park lamp contacts may corrode. This corrosion could cause the park lamps, head lamps and/or instrument panel lamps to flicker or become inoperative. This could affect driver visibility, increasing the risk of a crash. Dealers will reflash the software in the body control module.
Even without corrosion, bad power supplies can cause aberrant behavior.
Chevrolet Corvette 1997: On certain passenger vehicles equipped with electronic column lock systems (ECL), when the ignition switch is turned to LOCK, the ECL prevents turning of the steering system. When the vehicle is started, the ECL unlocks the steering system. The vehicle is designed so that if the column fails to unlock when the vehicle is started and the customer tries to drive, the fuel supply will be shut off so that the vehicle cannot move when the vehicle cannot be steered. If voltage at the powertrain control module is low or interrupted, however, the fuel shut off may not occur and the vehicle can be accelerated while the steering system is locked.
Defensive Programming
You've probably put error checking in your code to verify the inputs and take corrective action. This is generally a Good Thing, except when your system model gets out of sync with reality while traveling down the road at a pretty good clip.
Honda EV Plus 1998: The program logic for the motor electronic control unit (ECU) can mistakenly detect a failure of an electrical current sensor at speeds above 50 mph. If this occurs, the ECU quickly reduces vehicle speed in order to maintain proper feedback control. This can result in a sudden loss of power and unexpected deceleration that could cause a rear-end collision. Dealers will upgrade the ECU.
Putting a human in the loop complicates error correction. This cure resembles the previous problem.
Saturn EV1 1999: Electric passenger vehicles and pickup trucks containing a nickel metal hydride (NiMH) battery pack. These vehicles contain software with the battery pack module intended to disable the vehicle from being driven if the electrical resistance with[in] the battery pack exceeds a prescribed value. Affected vehicles will illuminate the "Service Now" and "Reduced Performance" telltales and continue to function in a reduced performance model instead of being disabled. If the telltales are ignored and the affected vehicle continues to be driven, heat could build up and a fire could result. Dealers will install revised software in the battery pack module.
Most people wouldn't drive very far in first gear but, in an X5, you might not notice.
BMW X5 2001: Over an extended period of time, abrasion particles could deposit on the electrical contacts within the park/neutral start switch, causing a disruption of the electric current through the contacts. If the disruption occurs when the vehicle is stationary with the transmission in the Neutral or Park positions, the transmission will switch automatically into the emergency program, which is displayed in the instrument cluster via a warning lamp and a message, "Trans. Fail-safe Prog." In this case, the vehicle can be driven in lower gears only with limited engine output. If this problem occurs, the ability to maintain speed in traffic and/or accelerate could be greatly diminished, possibly leading to a crash. Dealers will replace the transmission gear position indicator switch.
Ferrari cars have never been noted for their mechanic-friendly designs, but software introduces more complexity.
Ferrari 360 Modena 1999: A software programming error exists in the integrated electronic instrument cluster. This error exists in the brake system warning light monitoring function of the instrument cluster. The red electronic brake distribution warning light does not illuminate. As a result, if a brake problem were to exist, the driver would not be aware of it. Dealers will remove the instrument cluster from the vehicle and download updated software into the instrument cluster.
Mechanical Interactions
Automotive embedded systems, almost by definition, produce outputs that control mechanical actions. Thus, unlike ordinary software applications, embedded programs can be tripped up by real-world physics.
Lexus SC400 1992: The antilock brake/traction control computer logic program was incomplete, resulting in the trapping of brake fluid pressure in the rear brake lines. Failure to release the rear brake line pressure causes a condition where brake pedal application results in activation of the front brake only, and continual rear brake drag. This results in early wear of the rear brake lining, increases of stopping distances, and possible loss of vehicle control and a possible accident. Replace the defective antilock brake/traction control computer.
The model can get out of sync with the world in many ways, often because different rules apply in different conditions.
KIA Sedona 2003: On certain passenger vehicles, there was a programming error in the antilock braking system (ABS) electronic control module logic. The programming error could cause reduced braking force at speeds below 25 mph, which could result in increased stopping distances. Dealers will reprogram the control module.
This one seems to mean the brakes just stop working for a while.
Cadillac SRX 2004: On certain passenger vehicles equipped with all wheel drive, the antilock brake system (ABS) may improperly temporarily isolate the driver from the foundation brake system for a maximum of 1.25 seconds. This can cause in an increase in braking distance, which could result in a crash. Dealers will reprogram the ABS traction control system controller.
Antilock braking is one thing, but steering-by-wire also poses some interesting problems.
Pierce Custom Chassis 2002: On certain Custom Chassis equipped with the all wheel steer (AWS) 2002 system, the vehicle interface mobile program has a problem that causes one of the steerable axles to lock in the steered position rather than returning to the center. If this condition occurs, the driver can lose steering control, possibly resulting in a vehicle crash. Pierce is sending a factory technician to all vehicles and will upgrade the AWS system to eliminate this defect.
If you've ever wondered how much error checking is required in a safety-critical design, ponder this one.
EMC DS-2000 various years: Vehicles equipped with the EMC DS-2000 primary driving control system. There have been reports of vehicles equipped with the EMC DS-2000 [that] drifted or veered off course and did not respond to driver's steering efforts to correct the reported drifting. The driver can lose steering control, possibly resulting in a vehicle crash. A modification has been made that frees up the A2D [analog-to-digital] converter and feeds it with 1/2 the encoder reference voltage through a resistor divider. The software will be changed to monitor that voltage at startup to make sure it is an acceptable range. If during operation it changes from its startup valve, the system will give an error, switch sides and go into open loop mode of operation. This effectively test[s] all three references in the system.
Fire Control
The notion of putting pyrotechnic devices in every vehicle always seemed strange to me, but it's working out quite well. The overall hardware and software reliability required to make air bags work, first time, every time, with no false triggers, is truly amazing. Nevertheless, sometimes things can go wrong in the most peculiar ways.
Geo Prizm 1997: If the air bag computer in a subject vehicle experiences a mechanical shock (i.e., rapidly moving the front seat back against the stops or a sudden release of the parking brake) within a very short time after the engine is started, the air bag can deploy inadvertently. Dealers will repair the air bag computer.
I suspect the only way to trigger this one is to drop the seat on the control module while you're trying to manhandle it out the door.
Dodge Ram 1998: If a heavy object impacts the metal casing of the air bag electronic control module (AECM), located under the driver seat, the module could inadvertently deploy the air bags. Unexpected air bag deployment can result in occupant injury. Dealers will replace the AECM with a module that incorporates revised hardware and software on all affected vehicles.
The false-triggering problem becomes more acute as more sensors feed into the decision. In this BMW, the software model comes completely unstuck from the real world.
BMW 323i 1999: These vehicles are built with a side air bag system consisting of door-mounted thorax air bags, a head protection system (HPS) for front occupants, a central electronic sensor and diagnostic system, left and right satellite impact sensors, and associated wiring. This system is unduly sensitive to certain noncrash impacts, such as contacting large potholes or curbs at substantial speed. This could cause the side air bag and HPS to deploy without an actual side crash. In addition, the battery safety terminal (BST) would also activate, disconnecting the starter cable from the battery. In this case, the vehicle electrical system would continue to operate and the engine would continue to run, but after stopping, could not be restarted. Dealers will reprogram the central computer control module that regulates the performance of all of the vehicle's occupant protection systems.
Just deciding when an accident has occurred is difficult, but designers must now calibrate the air bag's deployment and power to the size of the person in each seat. Once again, the model can get out of sync with the world.
Hyundai Elantra 2004: On certain vehicles, if the front passenger seat is unoccupied or is occupied by someone lighter than 56.4 pounds, the passenger air bag is deactivated and will not deploy if a crash occurs. However, the occupant classification sensor may incorrectly interpret that a seated adult is an infant seat and deactivate the passenger air bag if someone heavier than 56.4 pounds who is seated with one leg resting on the outboard or inboard edge of the seat; with legs spread; or positioned off-center toward the center console and with his or her arm leaning on the console. In the event of a vehicle crash, the air bag may not deploy, possibly resulting in serious injury to the right front passenger. The occupant classification sensor will be reprogrammed.
Showstopper!
That familiar software-project term takes on a whole new meaning in automotive embedded systems.
Jaguar S Type 2003: On certain passenger vehicles equipped with automatic transmissions, with the forward drive gear selected and the vehicle stationary or in a forward motion, the gearbox could potentially select reverse gear, without indication. This failure could result in a crash. Dealers will reprogram the transmission control module.
It gets even uglier on two wheels.
Harley-Davidson 2000-2001: Ignition modules used on certain Twin-Cam 88 Screamin' Eagle ignition module kits and sold as dealer installed accessory items. The modules were manufactured with a software fault that could allow the module to shut off without warning. A sudden loss of power could cause a loss of control of the motorcycle and result in a crash, possibly resulting in serious injury or death.
Reentry Checklist
These are just the official recalls, so there are many exquisite little inconveniences that never reach the NHTSA database. Most of the time, embedded code works perfectly, but we must certainly figure out how to do better before we replace steering wheels and ballot boxes with software.
The NHTSA recall database is at http://www-odi.nhtsa.dot.gov/cars/problems/ recalls/recallsearch.cfm. You should consult the official, unedited version before reaching any conclusions.
You can download the database as 51,000 COMPLETELY UPPERCASE, tab-separated, text records file from http://www-odi.nhtsa.dot.gov/downloads/list.html. Some grep-fu will extract the interesting records, which you can then import into a spreadsheet for analysis.
Rust Never Sleeps is, of course, a 1979 Neil Young album. Frank Zappa referred to bromidrosis as an "exquisite little inconvenience" in Apostrophe, 1974. The Talking Heads put out Stop Making Sense in 1987. Rock on!
DDJ
