Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Channels ▼
RSS

Tools for Embedded-Systems Debugging


MAR93: TOOLS FOR EMBEDDED-SYSTEMS DEBUGGING

Emulators and logic analyzers can be a low-level programmer's best friends

Chris is engineering manager at Triage Corp. and can be contacted at 9900 SW Wilshire, Suite 250, Portland, OR 97225.


Referring to application programming as "high level" leaves those of us who twiddle bits for a living with the rather unfortunate title of "low-level" programmers. Working in an alternative reality where software meets hardware, we face a class of problems that would mystify application programmers. The title not withstanding, our work is not lowly--just misunderstood.

Programming anything from a simple microcontroller to a new compute system begins with the need to control hardware. Programmers who work in assembly language are typically concerned with devices that push that hardware around. Debugging software at this level often means looking into registers where each bit has a dramatic impact on system performance. It may involve integrating I/0 functions that require more speed than the system's designer allowed for. Or it can mean that we test a device driver the sole purpose of which is to chew up five critical days in a development schedule that was already unreasonably short.

Most frequently, low-level debug means probing software interactions using poorly understood application software while running on top of flaky hardware--and proving, if at all possible, that the problem isn't yours.

Theoretically, low-level debug should happen early in the development process, before everything else gets layered on the system. In a perfect world, every project would implement a well-thought-out design created using a structured methodology. The design team would sit down and design the hardware/software interface completely. In reality, it's more common to end up desperately pushing a probe while your cigar-smoking manager leans over your shoulder, screaming, "Why the hell doesn't this thing work?" Low-level software development is the foundation of all that is software, and if it's not right, everything above and below may be in serious trouble.

Life is easier if the software and hardware teams understand each other's jobs. Many companies working in the fast lane with the newest technologies find that success requires that engineering teams work closely together on a project from cradle to grave. This is essential, recognizing how tough the low-level environment is: System clock rates can be absurdly high or components can be A or B phase parts whose masks haven't even been fully verified.

Life outside the fast lane isn't much easier. Embedded code for manufacturing applications must often be written in environments that are, to say the least, barbaric. More often than not, the engineering budget is only enough for one or two software-development PCs that get used for everything from creating small tasking matrices to overly optimistic schedules. To make matters worse, some cross-development, debug, and analysis tools are too great an investment for many departments, meaning the programmer has to do without.

In this less-than-perfect world, someone actually has to isolate the problem and make the "simple fix," working only with the available equipment. Like plumbers walking into a house with pipes backed up beyond Tidy-bowl's reach, programmers require special tools like microprocessor emulators, microprocessor on-chip debug circuitry, and logic analyzers can make this job easier.

More Details.

In-circuit Emulators

In-circuit emulators are valuable for gaining insight into low-level software behavior. They can be simple to install, with one insertion point at the microprocessor. In-circuit emulation is great for loading a program, stepping through the software, exercising a target system, or recreating a fault, and can tell you what the microprocessor's registers are doing and whether the parameters are correct. Because they often provide a clock and some useful memory to work in, emulators also allow development of firmware long before the actual hardware becomes available.

For instance, I once helped develop an intelligent Z80-based I/0 card. Not surprisingly, the hardware specifications on the card were in flux when I joined the team, but the specification for the software seemed firm. Fortunately, I had access to a Z80 emulator capable of running on its own that provided memory into which I could load my program. I was able to begin implementation and debug my software long before the hardware guys had sorted out their specifications with the customer.

Unfortunately, device-level debug with in-circuit emulation introduces electrical loads that change the basic characteristics of the system itself. I've seen major software-implementation cycles run using emulators, only to find that once the emulators were unplugged and replaced with the stand-alone devices, the systems crashed. While emulator manufacturers try to reduce the amount they interfere with a system under test, electrical and timing intrusion is inevitable in emulator operation, particularly when system-clock rates begin to crowd the emulator's ability to keep pace.

A similar option is an evaluation board that runs off of a PC using the vendor's development software. Purchasing such a system can give you a complete development environment in which assumptions, software designs, and implementations may be checked out.

More Details.

On-chip Emulation

Because microprocessor clock rates keep on increasing and chip designers keep adding more gates to their designs, it's become harder to build a traditional emulator. Cache information is unreachable, intermodule bus interaction is unfathomable, and out-of-order opcode and data fetches and writes make software debug increasingly difficult. Fortunately, alternatives are emerging.

One approach is on-chip emulation and the JTAG (IEEE 1149.1) specification, an IEEE interface specification developed by an industry-wide committee. Currently, the major JTAG entries are the Texas Instruments TMS 320C40 and Motorola DSP96002 digital signal processors and the AMD 29200 microprocessor.

How do they work? With the right drivers to a development system and the right probe, on-chip emulators let you look at registers, step through processor cycles, or make the microprocessor do everything an emulator can do--in real time and at native-processor clock rates. On-chip emulation is not electrically intrusive--a major advantage when working at high clock rates. It also helps you get around the problem of what's going on in the on-chip cache or any of the on-chip peripherals such as I/0, shared memory, or floating point. Coupled with an oscilloscope and timing analyzer, on-chip emulation is every debugger's dream come true. For more details, see the accompanying text box entitled, "Inside JTAG."

The current limitation of the technology is the availability of software-development tools and JTAG-supported processors. Still, JTAG-compatible chips may lead to a probing approach that can keep up with the clock speeds and shorten debugger support-tool development cycles.

Logic Analyzers

Logic analyzers, the workhorse of assembler-level software developers, isolate software problems by providing a more complete view of the hardware/software interface and device-level behavior. A logic analyzer can combine software disassembly with timing information and a way of viewing simultaneous events on different parts of a system. It can also offer other types of data acquisitions from the same platform, making it possible to probe a microprocessor's bus, track other signals on the board, look at software performance and at oscilloscope signals--all at the same time. Logic analyzers can be configured to include digital oscilloscope functions, software-performance analyzers, and pattern generators. Using these features in concert with the traditional bus and timing acquisitions allows low-level software developers to acquire and interpret system information and behavior in new and unique ways. The text box, "Using Logic Analyzers" describes the use of logic analyzers in more detail.

Perhaps the biggest disadvantage of a logic analyzer is the sheer volume of information it produces. I've seen cases where logic-analyzer users made incredibly large bus and high-speed timing acquisitions, only to spend four days trying to understand what they captured. Smaller acquisitions usually do the job more quickly by enabling you to understand each step as you debug.

Another disadvantage is that logic analyzers typically provide no active means of controlling the target system. Plain-vanilla logic analyzers are passive. Others come with some form of explicit microprocessor controller, using ROM-emulation technology and requiring few system resources to provide emulator-like capability. Once the logic analyzer is configured, the user can hook directly into the system under test, download software and data, single-step through code, read or write the register set, and read or write any physical or virtual memory. These features are implemented by inserting a small amount of monitor code residing in system memory or EPROM. It allows the system microprocessor to execute in real time and maintains the use of the other logic-analyzer tools.

Beating Bytes into Submission

Real-world examples best demonstrate the power and limitations of the toolsets available to the assembler-level programmer.

For example, I recently confronted a D/A that was not putting out the desired signal. The hardware guys swore that all was well in their domain, and the applications-software team insisted that nothing was wrong in their lives. I was volunteered to settle the dispute.

Using a logic analyzer configured with the ability to control the microprocessor, I downloaded various versions of software and watched its execution. From the logic analyzer's keyboard, I modified some software versions to use different parameters or software functions and reexecuted the system under test. I even saved a few of my changes for later use and review. Using this approach, I was able to quickly prove (to the delight of the software team) that the code was driving the D/A properly.

But the problem still wasn't solved. By using the timing analyzer, the logic analyzer's built-in digital oscilloscope, and cross-correlated, time-stamped displays, it was possible to generate a great amount of data. I watched the various devices and their outputs, then compared this information against the logic analyzer's scope. The time-stamp information allowed me to cross-correlate events to a very narrow time-slice. Thus, I was able to demonstrate that the real problem lay somewhere in the D/A circuitry--completely independent of the microprocessor.

At the opposite end of the spectrum, I was once asked to force a solution very rapidly. A system needed to be developed that demonstrated the capabilities of a potential new product. The microprocessor was brand-new, hellatiously fast, and had very little software-development support. All I had were a few code fragments someone else had written, an assembler, a tenuous method of loading the executable into the system, and very little time.

As I began using the existing software, I quickly realized there was no means of seeing how functions were being executed. Several on-chip caches were being used, and it seemed impossible to extract the information I needed. While ranting and raving against the injustices of engineering life, I grabbed a logic analyzer and began reviewing the microprocessor's user's manual to see how exceptions were handled.

I discovered that as an exception was being taken, several useful pieces of information were forced onto an exception stack, including the address where the exception was taken and the exception type. I verified that I could disable the caches and force data reads and writes to off-chip memory. I also noted that even though there were no explicit software breakpoint instructions, there were several meagerly documented illegal instructions.

Armed with this and the logic analyzer, I built an exception handler the sole task of which was to turn off the caches and write stack and microprocessor-register information to off-chip memory. In this way, I could use the logic analyzer to trace and display the information my handler provided. The exception address information would let me know which illegal instruction had been executed, and the register dump would allow me to trace function-parameter information. The most important thing about this approach was that I suddenly had access to any part of that otherwise unfathomable microprocessor. I then assembled the new exception table with the address of my exception handler and inserted illegal instructions at different locations. I was then able to rapidly map the existing software and its functions.

This approach is valuable for extracting information when the only available tools are a logic analyzer and the microprocessor of the system under test. Any microprocessor may be probed in this fashion.

Bringing All Weapons to Bear

The world of low-level software development is a messy place to work. Problems range from application-software failure to hardware test and design verification. Low-level programmers need to use all available weapons in order to get out of desperate situations.

Emulators are wonderful, if you don't ask them to do things they're not designed for. They can transfer executable code and step through instruction sequences, but emulators have a hard time correlating different events on the same board, and they typically can't handle a system being developed at the highest possible clock rates. Logic analyzers provide a bigger and better picture of what's happening at the software/hardware interface. Used in combination with digital oscilloscopes, high-speed timing, and software-performance analyzers, logic analyzers help cross-correlate data to create sophisticated views of a system and its behavior. When combined with software breakpoints and monitors, the logic analyzer becomes a powerful, low-level software-analysis tool. Ultimately, there is hope that more microprocessor manufacturers will provide access to on-chip information, either through JTAG or some other yet-to-be-discovered approach for low-level software debugging.

No matter how loudly we proclaim that "It's not the software driver's fault," low-level programmers are usually the ones stuck between the rock and the cigar smoke. Finding the real culprit and saving our good name requires the right tools and a little thought.

Inside JTAG

Tony Coomes, Andy Fritsch, and Reid Tatge

The authors are engineers at Texas Instruments and can be contacted at P.O. Box 1443, Houston, TX 77251-1443.

As computer components and circuit boards become increasingly integrated, a problem arises. How do you test, validate, or verify the functionality of a specific output, function, or feature? Traditionally, this has meant connecting the system under investigation to oscilloscopes, logic analyzers, and microprocessor emulators. But as functions join other unctions on the same piece of silicon, the ability to test each function individually becomes lost. For example, devices such as DSPs with on-chip memory, peripherals, caches, and internal buses for interconnect--as well as ASIC devices with core logic--can have up to 300 pins with almost inaccessible internal connections.

Packaging advances such as surface-mount technology also contribute to significant test problems at the board level. With surface-mount packages, nodes must be accessed from the component side there may be components on both board sides. This presents problems for bed-of-nails testers. Additionally, the development of standard automatic test equipment (ATE) fixtures to test surface-mount boards has been so difficult that something else is required.

As a result, physical-device probing is becoming impractical, and testing is becoming more expensive. For this reason, considerable effort has been invested in developing techniques that allow devices to test themselves.

To begin to address the concern of system and component testability, the Joint Test Action Group (JTAG) was formed. Members include Alcatel, AT&T, Digital Equipment, the Department of Defense, Hewlett-Pachard, IBM, Philips, Siemens-Nixdorf, Texas Instruments, Unisys, and others. As a result of their efforts, there's now an international organization (JTAG) and its specifications are IEEE standards (IEEE JTAG 1149.1).

Building devices to meet the JTAG specifications requires that the device have the ability to test itself (called "built-in self-test," or BIST). The hardware built to perform this function allows access to circuits and nodes that cannot be observed, controlled, or emulated with even the most sophisticated ATE fixtures. This ability is based upon a test-access port and a boundary-scan architecture. A test port allows control and access to a boundary-scan capability and other test functions on the device.

For example, the JTAG serial-test access port (TAP) on Texas Instruments' TMS320C40 (C40) DSP chip is the pathway to on-chip emulation and the means by which TI's XDS510 emulator (a PC card) and the C40 communicate. A cable runs from the PC card to an active buffer pod. A cable links the pod to a 14-pin JTAG connector attached to the DSP under examination.

JTAG and the boundary-scan methodology allow an internal view into a DSP so that it can be emulated directly through a TAP. Traditional emulation features, such as displaying/changing registers and memory, single or multiple stepping, and running/halting are handled by the XDS510 through this port. These ports can be connected so that a system like the XDS510 can look at the inner operations of multiple C40s simultaneously. Thus connected, application software executing on these systems can be simultaneously debugged.

This is accomplished by designing access to internal nodes of the DSP through the test-access port. These internal nodes are said to be "on the scan boundary," and their state can be read out through the JTAG port. Boundary-scan technology allows a chip's internal digital logic to be tested without being probed. All that's needed is on-chip standardized test/scan circuitry.

Certain mainframe manufacturers have been using the boundary-scan concept for years to test their complex systems. The functions were not implemented at the internal chip level, but at the computer-subsystem level. The boundary-scan idea is not new; what is new is the acceptance of a standard (JTAG) for boundary scan and the test/scan circuitry on the chip. Now test-equipment vendors, semi-conductor manufacturers, and software/hardware toolmakers can design common devices that work together.

To implement boundary scan, every chip to be tested must include test/scan circuitry called "boundary-scan cells." The cell is implemented between each chip pin and the circuitry to which it is connected. In addition to the connection to the package pins and the chip's working logic, the boundary-scan cells are connected to each other to form a shift-register path around the periphery of the integrated circuit (hence the name, "boundary scan").

During normal chip operation, data flows between the chip pins and its internal logic as if the scan cells were not there. In test mode, however, these cells are directed by a test program to pass data along the shift-register path. Additionally, data loaded into test cells can be used in place of data flowing into, or out of, the chip points. This allows both the chip's internal logic, and its external, chip-to-chip connections to be tested. With JTAG boundary-scan technology, test/emulation sequences can be put wherever needed.

Using these building blocks, a chip manufacturer allows the device user to build a testable, debuggable system. This is driven by increasing integration of device functions and features, which, in the near future, may well become a requirement.

Using Logic Analyzers

When first confronted with a logic analyzer, the octopus of wires, leads, probes, pods, and ribbon cables may be daunting. Connecting a logic analyzer to the system to be probed can seem to require a small army of people to oversee the task, while logic-analyzer user interfaces sometimes seem best suited for communicating with visitors from other planets. It's easy to see why the advantages and sophistication of logic analyzers get passed over for simpler but much less powerful instruments. A valid question for programmers new to embedded-systems debugging may be, "Is it worth the effort?"

The real answer depends on what manner of bugs you intend to do battle with and how much of your system you choose to validate. For general embedded-systems development/integration/test/validation, no other debug tool is better suited.

Embedded-systems development most often requires:

  • Validating device timing and functions.
  • Validating exception handling and I/O functions.
  • Validating device events against embedded software commands.
  • Reviewing debug data from different points of view.
Logic analyzers have been designed to perform these exact functions (and more). They do this by providing device timing and microprocessor state tools. Each tool has one or more displays that the user can use to review the results of data acquisition. The other two major debug tools used in embedded-system development are the oscilloscope and the microprocessor emulator. In contrast to the logic analyzer, the oscilloscope provides only a couple of channels of debug information; the microprocessor emulator provides good software-debug capabilities, but little or no hardware information. A good logic analyzer provides the best of both, giving the user state and timing tools. Some new logic analyzers even provide digital oscilloscope tools, software-performance analysis, and embedded software load and manipulation tools. It's not uncommon to see logic-analyzer tool systems running state, timing, oscilloscope, and program-load functions simultaneously. By carefully choosing the logic analyzer functions and features, the learning curve for a logic analyzer is no greater than oscilloscope and microprocessor emulator learning efforts combined. This is illustrated in the following description of how to set up a logic analyzer:

  • Select a good general-purpose logic analyzer.
  • Select and connect the microprocessor probe made for your system's microprocessor to the system to be tested.
  • Select and load to the logic analyzer the disassembler package that matches the microprocessor's instruction set.
  • Connect device-timing probes to various points of interest on the system to be tested.
  • Power on the system to be tested, select initial logic-analyzer trigger conditions, and start an acquisition.
Several good general-purpose logic analyzers are available. To keep setup time to a minimum, many manufacturers sell logic-analyzer packages that include the timing probes and leads, the microprocessor probe and leads, and the microprocessor disassembler package best suited for your specific system. Logic analyzers configured in this way are nearly plug-and-play, thus making it relatively easy to immediately begin debugging the system.

Take the first several logic-analyzer data acquisitions to learn about the logic analyzer and to understand some of the debugging possibilities. Display the timing information in waveform as well as in bit form (1s and 0s). Review the software acquisition as disassembled data in its various forms (some logic analyzers let you organize assembly opcodes and data in different ways) as well as raw 1s and 0s (basic state-displayed information). Compare the disassembler display against your assembly-code listing. Watch the program flow as the logic analyzer has captured it. Then look at isolating hardware events based upon software commands and vice versa (that is, look at what happens when the system being tested handles a microprocessor exception). Following this procedure will help you understand how data is gathered and displayed. You can then move confidently onto more complex probing situations and trigger setups as you continue to learn and begin to debug your system. --C.P.

FOR MORE INFORMATION

EMULATORS                               Background Mode Emulators
Ice Master-PE                           68332/68340, 68HC16
8051                                    Embedded Support Tools Corp.
MetaLink                                10 Elmwood Street
P.O. Box 1329                           Canton, MA 02021
Chandler, AZ 85244-1329                 617-828-5588
602-926-1197
                                        8051 In-Circuit Emulator
EZ-ICE                                  8031/32/51/52
8031/32/51/52 68HC11                    Vail Silicon Tools
AMS Inc.                                692 S. Military Trail
160 SW Third Street                     Deerfield Beach, FL 33442
Pompano Beach, FL 33069                 305-570-5580
305-784-0900

AN196-MC Emulator                       LOGIC ANALYZERS
Annapolis Micro Systems Inc.            GPX Logic Analyzer
190 Admiral Cochrane Drive              Tektronic Inc.
#130                                    P.O. Box 4600, M.S. 94-86
Annapolis MD 21401                      Beavertown, OR 97076
410-841-2514                            800-426-2200

NICE-51                                 ML4400 Logic Analyzer
8051                                    American Arium
Tribal Microsystems Inc.                14281 Chambers Road
44388 S. Grimmer Blv.                   Tustin, CA 92680
Fremont, CA 94538                       714-731-1661
510-623-8859
                                        HP 1660 Logic Analyzer
EMUL-PC                                 Hewlett-Packard
805I 68HC11/16, 68332                   P.O. Box 58059, M.S. 51
Nohau Corp.                             Santa Clara, CA 95051
51 E. Campbell Ave.                     800-452-4844
Campbell, CA 95008
408-866-1820                            PM 3585 Logic Analyzers
                                        John Fluke Manufacturing
MICE-V                                  P.O. Box 9090
80C186, 68000/10/30, 68302              Everett, WA 98206
Microtek Inc.                           206-347-6100
3300 NW 211th Terrace
Hillboro, OR 97124
503-645-7333                            JTAG
                                        JTAG IEEE 1149.1
UEM Emulator                            IEEE Standards Office
Z181                                    445 Hoes Lane
Softaid Inc.                            P.O. Box 1331
8300()() Guilford Rd.                   Piscataway, NJ 08855
Columbia, MD 21046                      800-678-IEEE
41()- '90-7760
                                        TMS 320C40
Emulator/Analyzer                       Texas Instruments
8051, 68HC11, 80C196, Z80,              P.O. Box 1443
6805,68000, 68302                       Houston, TX 77251-1443
Orion Instruments
180 Independence Dr.                    DSP 96002
Menlo Park, CA 94025                    Motorola
800-767-4666                            3501 Ed White Blvd.
                                        Austin, TX 78721

Zaxpax                                  AM 29000
6800,80C186, V50                        Advanced Microdevices
Zaxtek                                  P.O. Box 3453
42 Corporate Park                       Sunnyvale, CA 94088-3000
Irvine. CA 92714
714-474-1170


Copyright © 1993, Dr. Dobb's Journal


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.