With us today is Paul Henderson, vice president of device test at Wind River.
Q: Paul, there are few better examples of real-time, embedded systems these days than a car. Unfortunately, it is starting to look like car-manufacturer Toyota's problem might have something do with software testing. So my question is this: Is testing real-time, embedded software like that in my Toyota Corolla different than, say, testing the software I use to gain access to the Internet?
A: There are several key differences between embedded software and that from desktop or Internet software that makes embedded testing challenging, including:
Fewer standards: Unlike the Internet world where custom applications are based on layers of standardized software, the embedded world has fewer standards. Often embedded applications are built from scratch. Even embedded operating systems have historically been custom-built in many industries. Companies tend to leverage commercial operating systems such as VxWorks or Linux and their associated middleware to save time in development and better leverage the work of others.
Greater processor variety: The dominance of Intel processors and the Windows operating system on the desktop has provided a fairly stable platform for app-builders, while the world of embedded devices is more complex. A greater number of processor types, compilers, and operating systems are in use across industries. In addition, with leaders moving to multicore processors and hypervisor-based architectures that can run multiple operating systems within the same device, the testing job is both diverse and highly complex.
Real-time constraints: While performance is important for Internet applications, timing is measured in seconds. In real-time systems, however, the timing is measured in milliseconds or microseconds. When embedded software is assembled and run at full speed on the actual device, new defects may surface as multiple processes or activities compete for the same resource. For example, you would not want a call to be dropped just because a text message arrived on your phone. These real-time interactions are difficult to test in every possible combination of events. The issues behind Toyota's recent troubles are not known. However, if the issue lies in software, there are a number of complex real-time process interaction scenarios possible.
Difficulty accessing internals at runtime: Embedded devices are like turtle shells " tough to get past the exterior to gain knowledge of the animal inside. Historically, device testers haven't had the luxury of easy-to-run, easy-to-observe and easy-to-interpret tests of software releases. They had to turn on and run the device while observing it from outside the "turtle shell." Testers and developers had only post facto analysis of the test case results with which to piece together what happened, what went wrong and why. This makes it difficult to observe and catch faults that may only occur internally and infrequently.
Q: Major software quality problems seem to be occurring with increasing frequency across more industries today -- and they are definitely happening more frequently in the embedded industry. What's going on?
A: I see several industry trends that are escalating the frequency and severity of software quality problems across the embedded industry, including:
- Skyrocketing software content: Devices currently have much more software inside that has to be tested, roughly doubling every two years.
- Architectural complexity: Rare is the old fixed-function device; the new norm is 32- and 64-bit architecture and multi-processors (and looking ahead, multi-core) running powerful operating systems, in addition to advanced user interfaces and networking.
- Compressed delivery schedules: Product development and test teams are being forced to deliver far more complex products in less time than ever before. Testing significantly more complex software in the same or less time can result in missed defects.
Q: Is part of the problem that companies seem to think at the outset that software testing costs too much?
A: Companies are generally aware that testing could comprise a considerable share of the total cost of a project, sometimes as much as 30-50%. They are also conscious that their device will never be absolutely bug free and are faced with a series of tradeoffs about how much they can spend and what level of bugs will be considered acceptable.
In an attempt to lower costs, during the past decade, many companies have moved testing overseas. However, companies are finding that simply lowering the cost of labor is not enough. It's becoming clearer that traditional testing tools and methodologies have become outdated. Due to the increasingly complex nature of embedded devices, a linear approach to dealing with geometric or even exponential increases in complexity is not going to be enough. People are still using old tools to do a new job, however, they "can't get there from here." Many are searching for a better, smarter way to handle device software testing today.
Q: Are the testing tools for real-time embedded systems up to the job?
A: A variety of tools are currently used for different test functions. Many tools today are designed for developers (e.g. unit testing tools, profilers, static / dynamic analysis) and are quite good. Companies need to focus on getting their developers to actually use them consistently. However, for system testing, also known as integration testing or functional testing, we find that existing tools are not good enough. Many companies build their own tools or use open source tools that have limited features. Most of these have been focused on traditional "black box" testing, which exercises and observes the device from the outside.
Wind River is investing in "white box" integration testing tools that offer testers and their managers real-time visibility into the device under test so they can make better decisions about quality status and focus their time and resources on what really needs testing. We have a new product, Wind River Test Management, which helps with this runtime test optimization. These tools are new and complementary to existing processes, and will enable teams to deal more effectively with device complexity.
Q: Assuming the industry can get a handle on software testing and quality, what's the next big obstacle for companies like Toyota?
A: I can't speak specifically to Toyota's situation. However, in general, many leading companies are looking to new iterative development methodologies such as Agile testing which use more frequent cycles to obtain earlier feedback on how they are doing with features and quality. These iterative methods differ from more traditional "waterfall" or "V model" methods, where testing was left until the end of the cycle.
An obstacle for embedded systems companies using Agile may be that the development and testing groups have historically been separate. They are often geographically dispersed with the test group in China or India. The testing group has great difficulty in understanding what code is changed or new in the daily or weekly interim releases that they need to test. Also, since they want to test on the device itself to really verify the software, it's very hard to know with black box techniques how thorough their testing is or if the device is really responding correctly. This is especially true for new complex architectures that have many asynchronous processes controlling multiple aspects of the device; increasingly, test groups are performing these on separate processors or multiple cores to reduce cost and raise performance.
In both of these areas, the white box testing tools from Wind River are viewed as a key enabler. These tools can report to Agile testers specifically what code is changed and can even recommend what tests to re-run to verify changes. They can also identify holes in test suites and probe the device at runtime to inject simulated faults and assure proper handling, or gather performance, timing and "forensic" data. This is done by better monitoring the device under test at runtime and better tracking what is actually being tested, and conversely, what has not been tested.
This embedded device integration testing requires continuous collaboration and will be an area of increasing importance that bridges both the development and test function. Implementing the new methodologies and weaving them into corporate cultures can also be a challenge, but it's far less expensive than convincing customers to come back after a big stumble relating to delivery product quality or safety problems.
