Cars and Software: Follow Up #2
Recall that automotive writer Wallace Wyss recently contacted Dr. Dobb's regarding the software side of today's cars, particularly Toyotas which have been getting a lot of (unwanted) publicity lately. Wallace knows a lot about cars, but not so much about software. So to find out more, he went to the experts -- you. Also recall that I promised to share with you some of the feedback that you and your peers graciously provided. I'll follow up with more response in a couple of days. Thanks to everyone who shared their 2 cents worth.
To provide context, here are the questions that were posed, followed by your responses. If you have any further thoughts, drop me a note at firstname.lastname@example.org or leave a comment.
1. I've heard that airplanes have tight firewalls among the different software/electronic components to prevent unexpected interactions among completely unrelated systems. Are there fewer firewalls in cars? Or can we not even call them firewalls? Is part of the problem that automakers already have a complicated car, then customers demand even more options -- say, Bluetooth or radar-controlled braking (as in Volvo) -- and the new options are being added into cars that weren't originally designed into the car's software? In other words, there are not good internal firewalls that can guarantee that spurious hardware/software problems from non-critical systems won't spill over into safety critical systems in a car?
2. Years ago, I drove a Camaro at GM's test track and the engineers telemetrically recorded every move I made and printed it out for me when I arrived back at the trailer. Why isn't it possible to instrument a car with two black boxes, one the stock Electronic Date Recorder and another one that would take and transmit a "snapshot" of what is called the system state space, i.e., the state of the key parameters that can tell you what was being executed at any point in time? I ask I keep hearing the glitches appear and disappear without a trace.
3. One engineer told me: "The more parameters you capture, the more you know. There is always a tradeoff in how much you can capture and the cost of capturing it that must be considered, too." So what if it costs $100,000 to capture and isolate the cause of unintended acceleration? Toyota has already lost $2 billion because of it. And they started this mess with over $20 billion in cash reserves.
4. If the different runtime system parameters used in each Electronic Control Unit were able to be captured and made available as well as the voltages/currents on the internal buses used to pass information, is that a likely area to find the glitch? I don't know what software looks like, but would excessive voltages/currents be visible like the squiggles on an earthquake point reading printout?
5. Why did automakers go to electronic controls of the throttle in the first place? I heard it was "so they could eliminate the hole through the firewall for the throttle cable" but that doesn't seem enough reason. Another engineer told me it has to do with meeting EPA regulations. In other words, the throttle system is designed to minimize emissions (90% or more of emissions are generated in the first two minutes of a car's cold start) as well as create as high a mileage as possible. Or is just because of it reduces weight, so the car gets better mileage?
6. Why do automakers want to go to electronic brakes or electronic assisted power steering? Again, because doing it electronically saves a lot of weight of the old fashioned mechanical/hydraulic hardware?
7. Toyota has been pretty secretive with their software, making it so other companies or even police can't download it. Do you think that even if they make the downloading machine for their Data Event Recorder available, if you didn't know the language and didn't the system design and operation extremely well, you might never be able to spot the source of the trouble?
8. Another engineer told me that what Toyota needs to do is to try to gather as much information as possible from each sudden acceleration incident and see if any common patterns emerge. Would that be the way to go from this point?
9. What about the latest theories in the press that cell phone signals or random spurious signals could be entering the car's electronic system ?
10. Does software in a car age? It seems in home computers after a while. glitches appear. I heard one writer say a modern car is only engineered to last 10 years. Shouldn't the software still be performing as designed in the 10th year? Of course I realize that if the hardware it executes on has a manufacturing defect or the wire bundles get frayed, the information the software program thinks it is seeing may get corrupted.
11. Being an old-fashioned guy (my first car was a '55 Chevy), I would like to see all the car companies go back to mechanical linkages on throttles, brakes, and steering. But another engineer told me that won't happen because automakers are under the gun to meet EPA and mileage requirements. Without electronic throttles, transmissions, brakes, etc., they just cannot meet ever-more restrictive government regulations. Automotive software writers say government regulation is driving them to use more software. Also, software driven systems are what allow manufacturers to constantly add new features to attract new customers, so software isn't ever going to go away. Am I dreaming and may as well ask Boeing to go back to mechanical linkages on the controls?
Responses to Question 10: My wife's SAAB developed a weird problem, when you tried to roll up the driver window (electronic, of course), the window would reach the top of the frame, then back down an inch. Simple, right? Either there was an overload and a safety system backed it up, or a bad overload sensor, right? Turns out that neither of these was the problem. The problem went away when the repair shop reflashed the software on the "door computer". Can automotive software age? Apparently so. I'm actually an embedded systems programmer. If I had to guess I'd say a power glitch in an inadequately protected embedded caused the software to get in a weird state. It could be that reflashing the software forced a hard restart, which is what really solved the problem.
Response to Question 1 I can speak specifically about software in cars, however, I will comment generally...Cars these days, I have heard, have upto 100 million lines of code. Programmers used to average about 1 defect per 10 lines of code. If that is still the case then that code mean 10 million defects in cars with upto 100 million lines of code. You can draw a conclusion from there. However, mission critical systems target 1 defect per 1000 lines of code. That would still mean a car with 100 million lines of code would have around 100,000 defects. If you go with cars built to 6 sigma, that means around 100 defects per 100 million lines of code. Given that cars these days are exploding with software, any new systems that have not gone through years of debugging will remain quite buggy. The software that has been around for a long time will have a minimum of defects. If a car uses Bluetooth, it is not a real-time protocol so it should not be used in systems that operate in realtime like brakes and acceleration as this presents an inherent design flaw.
Just because it can operate at 1 million bits transferred per second doesnt mean that it is real-time, merely that that is its maximum through put in its raw state. The same applies if standard networking protocols are used as they are not real-time either. Just because it feels real-time to humans doesnt make it real-time. Think about ABS and brakes. ABS pumps brakes faster per second than a human. Real-time to a human is 1 or 2 pumps per second, not the 30 or more that software can perform. This applies in general to any non-real-time protocol by extension. Why would a firewall even be needed unless the software is buggy? Each system likely operates independently of each other inside of a car, unless specifically engineered to work with other components like a combination radio/navigation system.
Most things in a car should be highly decoupled. When they are not they will introduce relationships that dont make any sense. Further even decoupling systems can cause inappropriate and unexplainable results unless there is a higher level management component that assigns priorities and/or enables/disables other components based upon other components actions/reactions. For example, if the user presses on the brake, should the accelerator be disabled or not? In non-computerized cars, the driver could both press on the brake and the accelerator at the same time if they so chose. There are probably critical situations in which pressing the brake should disable the accelerator, but there may be critical times that disabling the accelerator while breaking is a bad solution. Technically, it is not recommended to press on both the brake and the accelerator at the same time, but we're human and we recognize situations where it may be to our advantage to do both. However, as a software designer, it is hard to say when those situations arise and whether simultaneously pressing the brake and the accelerator makes sense....when do you prioritize on over the other? Unless the car can see, then it really can't attenuate the response.
So if it were me, I would automatically have the break disable the accelerator until the brake is no longer depressed.
Response to Question 2: This will happen eventually, but it is not done because this is not a simple thing to do. If it were, then all computers would have the software installed and allow us to review and figure out all of the problems. A black box is great from an understanding standpoint, but there should be specific rules that bar its usage against the passengers/drivers of the vehicle per law enforcement and insurance. It is none of their business unless you specifically authorize them to use it.
Response to Question 3: A device that captures this information would be useful and ultimately inexpensive if put into every new car. Retrofitting could be cheap too, but getting everyone to include it would be a chore and systems likely would not work with it. For such a device, you really need a generic solution which then means all components in the car should have to work to the common interface and now you probably need to introduce networking. Also keep in mind that you need hardware components that can withstand years and years of usage. You have to take into account extreme conditions per temperature both hot and cold, wet conditions, and other environment conditions that could deteriorate a solution. Then where do you put it?
Response to Question 4: All information is potentially able to be captured. It depends upon the software and the sensors to capture this information.
Response to Question 5: Moving from a carburetor to fuel injection to manage air flow and emmissions and what ever else was done partially because of laws requiring lower emissions, market forces demanding more performance as well as other demands. Cars today are seemingly just as heavy or heavier than cars of yesterday specifically because of laws demanding and the market demanding more and more safety equipment and safety solutions. Some solutions just cant be done without the aid of software or cant be done by minimizing cost of hardware.
Response to Question 6: Computer systems respond more quickly than purely mechanical solutions. Sure weight could be reason, but really the tolerances have more to do with response time of software systems vs mechincal systems. If we didnt demand so many safety systems in vehicles, we'd get better gas mileage because of lower weight and cars would cost less.
Response to Question 7: All companies will keep this secretive as they have little incentive to share information as they are for-profit companies and have no incentives to work together. Not being able to spot the problem or the source of a problem can be the case unless you can reproduce the problem.
Response to Question 8: Right now it seems to me that Toyota has a difficult time capturing the information necessary to diagnose them problem. If Toyota can capture all information then they should be able to find the issue.
Response to Question 9: I would hope that introducing other electronic gadgets that wirelessly communicate would not affect wireless systems in a car, but as an example, microwave ovens shouldnt affect wifi, but they do even though they dont necessarily operate on the same frequencies. This interruption of wireless services could easily happen in a car. It should not be done.
Response to Question 10: Hardware breaks down. What this means to software is that it needs to handle hardware failures and notify the driver and technicians that there are hardware issues. This means software needs lots of error checking built in which by design cant be tested until the hardware breaks down. Chicken and egg kind of problem....sort of. Software should not age, but the hardware does as it breaks down over time with usage and with time. They should be built with a specific long term hardware guarantees.
Response to Question 11: Mechanical solutions just cant deal with the sort of sub-second responsiveness that car systems have today especially as those systems related to govt/laws/regulations.
I forwarded this article/request to my brother, who currently works in this particular industry (and my wife, who used to).
Response to Question 1: If there are "firewalls" in cars, it is more that there are checks to make sure that data which is sent and received by safety-critical ECUs is not somehow corrupted at any point during the process of transmission and reception. This includes having special information sent along with the critical pieces of data, like checksums, rolling counters, etc. There are also systems in place to perform range and sanity checks on incoming data. However, any misbehaving module can bring the whole bus to its knees. The idea is to have fail-safes in place such that if the worst happens, the ECUs can perform at some minimal level of operation.
Response to Question 2: It is possible. But it is expensive (I'd like to know how much the system used at the test track cost), and the only benefit is not to the end user but to the car company itself when something bad happens. Will the average car buyer spend even a few hundred more dollars for something which will only help the police to figure out why the car killed them? Car buyers don't want to think that they are unsafe, and many (like myself) don't even like the idea of black boxes at all. If you make them pay more for it, it had better have some benefit beyond helping with the post-mortem.
Response to Question 3: The problem with intermittent problems is that they would have to have capture systems in all of their vehicles, and this would come at a cost much higher than $100,000. Just for the several million recalled cars, it could be in the billions of dollars to have systems like that, depending on what needed to be captured. Toyota has no idea which cars will actually exhibit this behavior, so spending a little bit of money to outfit just a few cars to test would be a complete waste of time and money. The only time the system is helpful is when it happens to be present and active when the misbehavior takes place.
Response to Question 4: I think that it could be very helpful. This technology is already used to calibrate and test individual ECUs. The problem is more that there is so much information to be captured. In order to capture time-coherent information from all of the ECUs in the system, especially if you wanted to see more than just the information which they are already sending on the vehicle buses, you would really have to have networks which were dedicated to nothing but this data capture. So many of these parameters change so quickly that the vehicle buses would be completely overwhelmed if even one ECU was sending such data as fast as it would need to in order to get all of the information that could possibly be useful in analyzing the problem.
Response to Question 5: I don't have a good answer for this one. It was actually a surprise to me that electronic throttle control is as ubiquitous as it is. When the fuel metering came under electronic control (fuel injection vs. carburetors), I think that it just made sense to do the same with the throttle. It certainly makes things like cruise control and electronic stability control easier because you can treat the driver commands as an input to an equation instead of an input which must be mechanically countered in some situations.
Response to Question 6: The answer is both a savings in weight and the complexity of hydraulic lines all over the place, as well as the constant load of the hydraulic pump on the engine and its effect on gas mileage. With electric power-assisted steering, you get steering assist which only creates a significant drain on the electrical system when the motor is working, not an all-the-time drain on the engine power like a hydraulic pump. There is also an advantage in that fancy features like auto-park can work with a system that is already under electronic control with few changes to the system. The other big reason for moving to electronic vs. hydraulic systems is the advent of hybrid and electric vehicles. In these vehicles, there is either no engine, or an engine which only runs some of the time. This makes it necessary to provide a different kind of constant power to these features, without being able to take advantage of the spinning motor that you would find in conventional gas/diesel vehicles. Features which are moving to electric for this reason are things like steering, braking, and the A/C condenser.
Response to Question 7: Toyota claims that the usefulness of the data which is captured in their black boxes is limited at best. Ford, for one, has been providing this information to law enforcement for a while now, but I don't know how helpful it has been in crash investigations. It certainly captures information like the vehicle speed, etc., but if it does not capture just the right set of parameters from the thousands that are there, it's just as useless as no special information at all. I think that it's doubtful that just looking at the numbers would be enough to find the issue.
Response to Question 8: That is certainly how I prefer to approach recurring failures. It's much easier to see a pattern than an individual data point. The problem here is just that the pattern may not be visible from the data which is available. If you don't have insight into all of the affected systems, and you are just relying on the testimony of a traumatized (or deceased) driver and the vehicle itself (which no longer exhibits the issue), the probability of discerning a reliable pattern is low.
Response to Question 9: Though I can't list the specifics, I've heard of some strange failures that come from EMI. It's not just the different communication buses that can be affected. It's possible even to flip bits in RAM, and the effects can be completely unpredictable when the state machines or other variables in the system are corrupted.
Response to Question 10: Leaving aging hardware out of it, if the software is well-designed, I would not expect more failures in the 10th year than in the 1st. The "aging of software" in home computers comes from the fact that the software and OS get updates and the computer gets bogged down over the years with junk. This is not a problem in the embedded systems in vehicles (other than the infotainment systems which get clogged up with your millions of MP3s). The OS and the rest of the software stay just the same. Newer models may be faster and have new features, and there may be updates to the software itself, but then all of the software is updated at once, and it is tested all together as a single piece. When you apply the latest patches from Microsoft, what is it doing to your other programs?
Response to Question 11: There is no chance of going back to the good ol' days of mechanical systems. However, it will be a long time before I could be convinced to buy a car with true "steer-by-wire" or "brake-by-wire". I think that these systems are still a ways out, due to the liability of having the computer be the only thing providing direct control. Electronic assistance is one thing. I'd prefer not to be taken out of touch of these two elements completely, though.
Response to Question 1: You would need someone from the design team of each manufacturer to answer that for certain. In the beginning, when there weren't "in car" busses like CAN to carry information from one computer (microcontroller) to another, there was the ultimate firewall between components -- no interaction. However, as time has gone on, a certain amount of integration has become necessary, and even desirable. A for instance: the transmission controller is integrated with the engine controller because it makes adjusting shift points based on the load on the engine, etc., possible. If you want traction control, now you need to interface the brake system with the engine management system, etc. Paddle shifters, etc., require even more control of the engine management system. As to the firewall systems, we would hope that each subsystem would validate it's input, but since that information is proprietary and most likely involves trade secrets, unless someone who works in those systems pipes up, we may never know.
Response to Question 2: It's entirely possible, and even probable that Toyota has done this. However, there's a difference between a glitch that affects every car every time, and something that is very intermittent, both in which cars it affects, and when the anomaly occurs. The data recorder GM used is probably very expensive and it's not practical to equip every car made with something that can log that kind of information, but it's that kind of information that might have lead to a solution faster. Reproducing a problem found in the field is one of the most difficult things people who design this stuff face. In the lab, you're working with a small sample, yet, once you reach production, there could be millions of these things made, and a simple process change made by a supplier could have a detrimental effect. There's also a privacy issue with collecting data from every car.
Response to Question 3: See the answer above. It's not the cost, it's the practicality.
Response to Question 4: It depends on the nature of the glitch, and it depends on the nature of the device and it's programming as to how it might reject erroneous input. In terms of logging, some glitches that could cause a part to "latch up" or misbehave in some other manner may be only a few nanoseconds in duration. It takes a pretty expensive piece of equipment to detect an event that short. Plus, the shorter the duration, the more often you have to sample. The more often you have to sample, the more data you have to store and/or process. Since cars are cost reduced down to pennies, equipping the car with more than a small amount of logging memory could be prohibitively expensive, and most of the microcontrollers used in the car are pretty busy making sure everything is running right, so you'd have to add something to do the analysis, and up goes the cost.
Response to Question 5: If you want an automatic transmission that has really smooth shifts, you probably want to modulate the throttle - traction control, modulate throttle and brakes -- paddle shifters, engine and transmission. On a prius, when you come to a stoplight, everything shuts off. When you take off, it takes off on electric if there's enough charge in the batteries. As we get more into hybrids, traction control, etc., all this requires coordination between systems at a level your '55 Chevy never dreamed of. Plus, replacing a linkage with a wire not only possibly saves weight, it gives the designer more flexibility in where those controls are placed. In a hybrid system, your throttle linkage would have to be integrated into the electric motor drive anyway, so you definitely are going to be controlling things electronically.
Response to Question 6: Could you imagine anti-lock brakes or traction control done mechanically or hydraulically? How complex that would be, and how heavy? Not to mention that it's relatively easy to make a software based system "adaptive", where mechanical ones are far harder. Remember how finicky Hillborn mechanical fuel injection was? Notice that you don't have to "pump" the accelerator before starting your car, and new cars start every time with no user input? This is all possible because the computer is checking the humidity, etc., and delivering exactly the right amount of fuel at exactly the right time, regardless of temperature, etc. Even at a cold start when the system is running open loop, it's still able to deliver exactly the right things the motor needs because it's less perturbed by things than carburetors, etc.
Response to Question 7: To a point -- there's a certain amount you could derive just by looking at the data and reverse engineering it, but it'd be a lot easier if the code and the data were made public -- or made available to the person(s) providing an independent third party analysis. However, there might be trade secrets and patents involved, plus, just taking the time for someone to understand everything may be longer than it took the designing engineers to figure out what the problem was.
Response to Question 8: To a point -- they should be looking at what data is possible to log given the hardware installed in the car, but there might not be enough data there to get any answers. However, if you can pinpoint it to a particular sequence of events, at least you'd know where to start looking.
Response to Question 9: Possible, but not likely -- most communications channels in use today have error correcting protocols, with retries, etc. That's pretty elementary. Also, stuff put in cars has to be pretty noise immune since an automobile is a pretty inhospitable environment already with respect to noisy electric motors, ignition systems, etc.
Response to Question 10: Improperly designed, yes, but embedded systems tend to be very tidy in that respect. Unlike your computer, there are very finite resources available, you're not installing new software all the time, and there aren't things like hard disks to get fragmented., etc. However, it's possible for a programmer to do things like fragment memory, etc., by not following best practices for embedded systems, even though this is not likely to be the problem. Anything this random and transient is going to probably be some improbable sequence of events, exposing a logic error, possibly including a hardware defect.
Response to Question 11: That's true -- Google the term "CAFE standards" and you'll see what the automakers are up against. Fuel economy has to be balanced with emissions and this is something that is just not possible without the tight controls engine management systems have over air / fuel mixture, ignition timing, etc. All this is controlled to an incredibly fine degree and the input of sensors in the system like the mass airflow sensor, vehicle speed sensors, O2 sensors, etc., all make sure the engine is running as efficiently as possible. Remember that not only is this an economy / emissions thing, but it's also possible to create 305HP V6 and 426HP V8 Camaros, etc., that make over 20 MPG on the highway. The old way was to make a great big hole in the top of the motor for air and dump enough fuel in it to make the car go fast. Now we have the ability to dump just the RIGHT amount of fuel into the system at JUST the right time, in just the right place, to make the kind of performance required at that moment in the most efficient way.
I've been involved in software design since 1969 and with automotive software since 2000. I work with body control software which involves lights and windshield wipers and tire pressure monitoring and keyless entry but does not include engine control or braking or airbags.
Response to Question 1: Firewalls -- what is there is mostly by default rather than by design. The only communication one ECU (electronic control unit) has with another is typically over the communication bus. The bus is usually CAN (controller area network). Unlike the Internet where messages are largely free form, automotive bus message are tightly formatted bunches of signals that indicate the current value of some sensor or indicate that some event has occurred. The communication stacks are constructed such that you only actually receive a portion of the messages that are on the bus, the filter can be either hardware or software. Validation of incoming signals is pretty much the responsibility of the ECU receiving the signal -- some people do a great job, some to a shoddy job. The fact that the messages are so strictly formatted tends to provide isolation between the ECUs -- none of the Internet style network vulnerabilities exist, the worst you can do is send incorrect data for a value.
Response to Question 2: Data recorders -- they still exist and if Toyota can find a way to recreate the issue on a test vehicle it would be a very good way to isolate the problem. One problem is that most automotive testing is done on the bench these days. Budgetary considerations have drastically reduced the number of test vehicles available for engineering use. Only really high-profile problems get access to test vehicles. Up until a couple of months ago, Toyota throttle issues were not high-profile.
Response to Question 3: Capture lots of data -- no argument except this also falls under "high-profile" -- capturing and analyzing lots of data takes manpower and money.
Response to Question 4: Capture runtime parameters -- a good idea, but likely no practical in production ECUs. Systems that do just this are readily available for use during ECU development. Only problem is they use system resources that cost money. For production you eliminate the data capture ability and switch to a less capable processor to save money.
Response to Questions 5 and 6: Why use electronic throttles, steering, etc. -- let's see there was COST, emissions, COST, fuel economy, COST, weight, COST and COST. People outside the auto industry have no idea what it's like to deal with a system where lower cost is the only real objective. If you can find a way to do something for less money it will likely be adopted no matter how odd or how crude the solution seems.
Response to Question 7: Toyota secrecy -- this is an industry problem but Toyota has it worse than most. The way patent law works makes it difficult to actually secure meaningful patent protection for intellectual properly. Secrecy is the result. And yes, if they made the code available, people would figure out how to use it.
Response to Question 8: Common threads -- I would suspect Toyota is hard at work trying to find a common thread. If this were my problem that would be the first thing I looked for.
Response to Question 8: . Cell phones, etc. -- definitely a possibility. While cell phones, MP3 players, etc. are pretty well tested, some of the cheap aftermarket accessories (chargers in particular) are hardly tested at all. We had a remote keyless entry issue 7 or 8 years ago tied to a particular brand of cell phone charge that had been brought it from China with no testing whatsoever.
Response to Question 10: Does automotive software age -- not really, it's usually stored in flash ROM. On the other had, the sensors that provide input to the ECUs do age, and so does the ECU hardware itself. Aging motion sensors are susceptible to noise, they can become nonlinear and their limits may change. For most ECUs, the one piece that defines the 10 year lifetime is the EEPROM. EEPROM is used to store user settings, trouble codes, serial numbers - any variable data that needs to survive a battery disconnect. Typical EEPROM life is between 1000 and 1 million write cycles. When you calculate ECU lifetime you divide the estimated write cycles per day into the lifetime of the EEPROM and make certain you exceed 10 years.
Response to Question 8: Mechanical linkages -- part of me agrees with you, but it ain't gonna happen. While you could possibly do it on a conventional car or truck, how would you do it on a hybrid with regenerative braking or on a full electric? For better or worse, these things are with us. Our job is to figure out how to design and test them to make the hardware/software package bulletproof.
For what it's worth, I suspect that when they are done, Toyota will find that some ECU is not properly validating an input from some sensor (likely the throttle position, but maybe something else) and deciding that the driver is intentionally asking for full throttle. We also need to look into why certain system level decisions were made, not only at Toyota but and VW and a number of other OEMs. In particular why they decided to give throttle priority over braking.
Response to Question 1:In a enviroment like a car engine, a firewall is also prone to glitches and noise like the other systems. Spurious hw/sw problems should be prevented, detected and silenced. Prevention is the best option -- I would suggest overengineer and using design/components much above requirements.
Response to Question 2: Glitches should be expected. Power supply and ground line noise would cause unpredictable problems everywhere, including the blackboxes. Noise in unavoidable, no matter how many blackboxes (the more, the worse).
Response to Question 3: I would rather to have a "emergency switch", to be used by the driver. You can have it under a protected lid, and require two buttons to be pressed.
Response to Question 4: Same as 3. Why spend too much on detection? Let the driver decide. This could become a safety standard to be used by any car brand.
Response to Question 7: Reverse engineer and sw reconstruction might make simulation and testing possible by third parties. I would not look for real data, but would try to make the systems misbehave in a lab environment.
Response to Question 8: I think I would do the same as Toyota.
Response to Question 9: Noise under the car hood, from engine and car systems is many levels of energy above that.
Response to Question 10: I wouldn't bet on aged software.
Response to Question 11: I currently own a 76 Puma. I like it, but fuel injection is much better for a daily drive.
In my opinion, the answer to these questions resides in both the hardware design of the controlling/monitoring units and the software (probably microcode or coding of programmable array logic). My background is electrical engineering. I have had a lot of experience in designing controllers and using feedback logic to monitor a state that is under control. Electronic devices can detect, measure and record a lot of things and software can be implemented to make many higher level decisions, such as initiating a gradual shutdown is key parameters are "out of bounds". But, and this is a big one, generally speaking hardware cannot be designed to detect and respond to instantaneous glitches or spikes in a signal that could cause things to go bad. Usually, filters are in place (capacitors) to reduce the effect of voltage/current spikes in analog lines i.e. bus supply voltage. But if somehow a digital signal spikes (I've seen this happen) it causes a logic failure and usually points to a bad component or a short/open in the PC board. You can design-in ways for the system to respond to some of these errors and reset or shut-down but in many cases it takes too much circuitry and would cost too much.
In 1998 I was on a team that examined banking industry software (COBOL, FORTRAN, Paradox, etc) written in various decades, in preparation for the turn of the century -- Y2K. The older code was more of a mess than the newer code, partly because there was less RAM to work with in the early days, so a storage address might have to be used for different purposes at different stages of the program, and partly because programmers in the 1960s and to a lesser extent '70s thought that wierdness was cool. If the team found code that looked troublesome, they rewrote those parts. Then we ran simulations for various values and dates, e.g., 31 Dec 1999, 01 Jan 2000, 02 Jan 2000, 31 Jan 2000, 01 Feb 2000, 28 Feb 2000, 01 Mar 2000, ... . As it happened, we used WinRunner, but that's not the only simulation program on the market. We had directed our simulation programs to write certain data into a file that we would examine later, and to generate report pages as if they were being printed on, e.g., 01 Jan 2000, by the software being examined. Then we examined the files we'd written and the sample pages we'd printed, to look for anomalies. Then we collected and reported our findings for every program we studied, and if we found no-no's, the software had to be remediated and retested. I think Toyota and whoever else is studying the problem of uncontrolled acceleration should use simulation as one method of locating the bug.
As is true of all diagnostic systems, what the car computer "sees" is a function of the quality of the probes that are used to capture state information and the amount of information the probe can capture and communicate. As a simple example, consider this situation:
I have a 2010 Honda Insight with a Hands Free Link and a relatively new Bluetooth capable LG Incite. After going through the necessary steps to set up both devices for the first time, the devices connected and I was able to place a call from my driveway. So I decided to drive around the area and place a free hands free calls. Once I was about a ¼ of a mile from my house, the Blue Tooth connection dropped. I pulled over and tried again. I got the connection reestablished and continued on my way. After about half an hour, the connection dropped again. This continued for the next hour as the connection dropped and I pulled over to reestablish it.
This problem caused me to visit my AT&T store and visit numerous sites including the Honda Hands Free Link site. No one had any clue why this occurred. I finally visited the AT&T Service Center. Their guess was that some of the phones had weak Bluetooth antennas. So they shipped me a new phone. I repeated the process. Same result. This time, however, I was able to study the screen on the phone as the connection dropped. Each time the connection dropped, I had moved away from one WiFi site to another and the phone displayed a message allowing me to select another site. Apparently, that process caused the phone to hang and the Bluetooth connection timed out. I changed the WiFi setting and everything worked seamlessly for a week or so.
Then if failed again. This time, however, I had to reinitiate the pairing process by creating another shortcut in my car's navigation system. It was identical to the other, but it connected while the other one didn't. Now I have four of these shortcuts, and each one will connect at various times for some unfathomable reason. This situation occurred with relatively stable devices and technology that has been in use for quite some time.
The point of this story is that the person who has to debug this process is the user who has incomplete information from which to work. I am certain these problems being faced by Toyota will become more widespread across the industry as more and more disparate systems are brought together without a coherent diagnostic architecture in place. I have worked in the software industry for more than thirty years and have yet to see anyone create such an architecture outside aeronautics, and even that industry has its bad moments.
More responses to come...