Building an OS-Aware Debugger

• Standard Monitor Debug Mode.

• System Debug Mode (SDM).

• Task Debug Mode (TDM).

To debug system problems, the Standard Monitor Debug Mode can be used without additional communication with the OS. The debugging of tasks can be done either in a System Debug Mode or in Task Debug Mode. In System Debug Mode, the whole system stops if the monitor gets control of a task. In Task Debug Mode, only the current task is stopped, but the monitor lets the system continue with any other tasks.

One solution to these kinds of problems is to use debuggers that are operating-system aware. With an OS-aware debugger, you can, for instance, access OS-specific structures and elements in an easy- to-view format, thereby increasing productivity. Without the task awareness of an OS-aware debugger, on the other hand, you must decode the raw data by hand -- a difficult, time-consuming, and error-prone process. OS-aware debuggers also let you view task-description blocks in a Task Window in their native format. This lets you immediately see the tasks that are, for example, blocked waiting on a semaphore. With nonOS-aware debuggers, you have to view the task-description block in a memory window as a series of bytes or words.

Similar benefits arise when looking at messaging structures, such as queues and FIFOs. With OS-aware debuggers, the debugger takes care of decoding all the links and structures and presents the data in a user-friendly manner. If your OS will support it, then your debugger can be used to set complex task-specific breakpoints, from simple ones that say "break on every task switch" to more complicated conditions, such as "break on executing printf(), but only if the SEND task is active." The benefits of these complex breakpoints are easily appreciated when you have a problem such as one task sending spurious data to the printf(). If you used a standard debugger, then you would have to examine every execution of the printf() to isolate the problem. With a task-aware debugger, however, you can isolate the tasks you suspect and ignore previously tested code.

There are numerous other occasions in which adding OS-aware features can either enhance the usability of the debugger or simply make the debugging task less onerous. For example, an OS-aware debugger can take advantage of the OS load functionality; tasks can be loaded by the debugger, allowing for an upgrade of the system while on line. You can use this load functionality to replace a faulty or inoperable task or incrementally add extra tasks to determine response time and processor burden.

Making Debuggers OS Aware

There are two different ways to make a debugger OS aware. The easiest approach is to make use of existing debug hooks in the operating system. This is the method we use in the XDB Debugger for pSOS from CAD-UL (http://www.cadul.com/; the company I used to work for). XDB connects directly to -- and speaks the language of -- pROBE, the standard console debugger supplied with the operating system. This method is also used in the Linux debugger I describe here, although some kernel patches and a user module were also required to get the necessary level of integration.

The second approach to debugger integration is used when there aren't any debug hooks for connecting to the OS. With this approach, you must add code to the monitor/debugger. This is the approach taken with the Real-Time Linux RTL_monitor. Essentially, RTLinux (http:// www.fsmlabs.com/) is a modification to the standard Linux kernel, which adds traditional hard real-time features (similar to the VenturCom add-in for Windows Embedded NT). For an in-depth description of RTLinux, see "Inside Real-Time Linux," by Jerry Epplin, DDJ, March 2000.

Linux thread debugging has forced modifications to the kernel source files. When a fork() system call is taken in a typical OS and a new process/task is created from the original task, then new code, data, and stack space are allocated and the original process is copied to this new memory location. To speed up the system, Linux operates a copy-on-write feature that defers the actual copying of a memory page until a request is made to write to one of the copies. This method can realize significant time savings because it avoids unnecessary paging of memory to and from the much slower secondary storage on disk.

Contrast this with the threads created using the clone() system call. When a thread is created, the same data and code segments are used as the original, but each new thread is given its own stack. If a process is being debugged and it already contains a software breakpoint, then it will be active in the next thread, but no breakpoint handler is active. Because the monitor is not normally aware of this new thread, the thread runs on its own and ultimately fails. To overcome this problem, the Cadulclone() function is added to the kernel to inform the monitor that a thread has been created, and that it must be attached to the monitor using the ptrace() system function.

If a breakpoint is subsequently requested in either copy after the thread is created, then the Linux copy-on-write feature lets one thread remain intact and the breakpoint is set in the other copy. Patched versions of the affected kernel files are supplied with the monitor source files. Figures 1 and 2 list the files modified in the kernel.

From simple round-robin schedulers to traditional RTOS with task switching and intertask messaging, most operating systems have either a subset of the available debug hooks or something that can be easily modified to provide these hooks. Fortunately, it isn't necessary to implement all the features in one session -- you add only the features you need.

Both implementations I discuss here are based on CAD-UL's XDB ROM Monitor debugger. XDB is a remote cross-debugging client, currently available on Win32 and Sun Solaris systems. This version of XDB works together with a protected-mode ROM monitor loaded on the target. A communication link between the target Linux system and the host is used for transmission of data from the debugger to the monitor.

In Figure 1, the RTLinux implementation, Linux is running as a real-time task along with other real-time tasks. Apart from the idle task, Linux is actually the lowest priority task on this system. This is a traditional RTOS with little in the way of debugger hooks. Listing One (moninit.c) handles much of the direct translation between the OS and monitor. For instance, the OSGetTaskID function returns the current task identifier by reading it directly from the RTLinux structure rtl_sched[0].rtl_current.

In contrast, Figure 2, which illustrates the Linux user_monitor, outlines a modified approach used when the OS supports some debug hooks. The user_monitor section of the monitor is run as a Linux process and uses the system calls ptrace() and ioctl() to interface to the OS information. There is a small user_ module that is added to the kernel to facilitate communication between the kernel and user space. Modifications to a few of the kernel files have been made to inform the monitor of a debugged program exiting or of the creation of a new thread.

When the GNU Debugger Just Isn't Enough

Linux system developers for x86 systems can use only the GCC tools to compile Linux because no other compilers are supported. When developing an application to run on Linux, it makes sense to use the GNU debugger (GDB) as well as GCC. However, these debugger system calls and hooks are available only on a working system. Thus, the first step is to get Linux running on the target system.

It is important to understand the differences between the development process required for Linux running on a typical PC and running it on an embedded system. Figure 3 outlines the basic differences between a desktop PC and a videophone in a visual format. Both are x86 systems, but the similarity ends as soon as you look at the hardware interfaces. Porting Linux to a known hardware platform is relatively easy, especially with tools from the likes of Red Hat and Caldera. However, while these vendors supply a list of supported motherboards, hard drives, and network cards, it is still likely that none of the devices on the embedded system would make this list. Thus, taking a standard PC and removing the display and keyboard and making a headless system is not for the faint of heart. Fortunately, companies that produce adaptation kits for even this scenario have sprung up. (Check out HardHat Linux from Montavista Software, http://www .montavistasoftware.com/ and Embedix from Lineo, http://www.lineo.com/, for supported versions of Linux that have been specifically designed for embedded development.)

Up and Running

Getting embedded hardware up and running to boot Linux requires patience and hard work. Although prototype systems might use a floppy disk drive, there isn't a hard drive to hold the Linux installation -- it must be loaded to the flash memory. This means that you can't use the standard IDE utilities that partition disks when installing Linux on a desktop. The next obstacle is the lack of support for the custom LCD display. This engenders problems with the BIOS, which assumes there is both a standard keyboard and standard screen. Due to these technical challenges, an emulator is usually required to debug the startup code. This was, in fact, the scenario for one of our large communications customers when I worked at CAD-UL, and it was the impetus for the RTLinux and user-level monitor development I describe here.

Source-level debugging of GNU-compiled and GNU-linked applications is available by processing the a.out file using the DBG2BD GNU filtering utility. This filter can process many flavors of x86 GCC output, including a.out elf/dwarf and coff/stabs++. Figure 4 outlines this process. To get source-level visibility in the debugger, the output from the GNU linker must be processed to convert the symbol table information into a format that the XDB debugger can load. The BD symbol file can also be loaded into any emulator version of the XDB debugger, thereby giving emulator support to any GNU user.

Multiple options exist for modifying DBG2BD's operation, belying the tool's original incarnation as a part of the embedded developer's toolset. For example, although I'm assuming the use of Linux here, the filter can process files for WindRiver Systems' VxWorks, or any operating system that uses the GNU x86 Compiler tools. The utility generates two files from the output of the GNU linker. The first, the BD file contains the converted symbol tables and is retained on the host side; second, the HX file contains the executable code and constant data. The BD file is a proprietary symbol file format used by all of CAD-UL's debuggers. This provides two options:

• You can load this file into the Linux or RTLinux XDB and connect to the monitor.
• You can load it into any of the x86 emulators that XDB supports to get hardware-assisted debugging. If necessary, the HX file can be downloaded directly to the target memory, using the debugger.

Once our customer's engineers successfully ported standard Linux to their custom target, they decided to stay with the same XDB debugger client for programming the RTLinux modifications and for application development. This enabled an easy transfer for programmers through different stages of the project, because only one debugger user interface had to be learned. There was also the added benefit of being able to plug in an in-circuit emulator if a difficult problem arose in the application development phase.

OS Function Calls and the Monitor

Our customer's engineers used CAD-UL's Protected Mode monitor to perform all the regular tasks any monitor is required to do. The monitor handles communication with the host debugger, parses the commands, and performs reading and writing of the target memory and registers. In addition, it can perform Run Control to start and stop execution and control breakpoints. However, normally it cannot handle OS-specific functions such as reading OS task lists or loading a task using the debugger. To enable these features and others, we've developed a standard mechanism to add OS extensions to the target monitor.

Listing Two lists the functions supported by the monitor and XDB. The interface between monitor and OS consists of a set of functions and monitor service calls (SVC) to exchange information about the system and the tasks. The SVCs are invoked by a user-definable software interrupt in a manner similar to the regular PC BIOS int13 and int20 interrupts. With this interface, a complete handling of debugging efforts for operating systems is possible. Because the interface and monitor don't contain any assumptions about the operating system, there are no restrictions on the operating system.

The interface to the operating system allows the display and control of tasks, task-specific breakpoints and complex, or OS-specific breakpoints. A pseudo Task Switch Selector (TSS) data structure, based upon the standard x86 TSS, passes information such as register values and flags between the monitor and the OS.

The execution control is split between the monitor and operating system. The operating system suspends and activates tasks as usual. The monitor tells the operating system if a task is in debug mode or not; it also tells the OS if it can be resumed to run on its own or if, for example, the single-step trap flag should be set.

Direct system commands can be used in the debugger by using the OS_command functions. This could be used to perform, for example, an "ls /proc"; the results can be returned to a window in the debugger, either directly or via the OS_Get_Output. The load functionality is extended to support OS-loadable files, which can be loaded by the operating system. The relocated address information is returned to the debugger, where it is used to relocate the debug symbol information.

A struct, (osCalls), is filled with pointers to OS-specific functions, as in Listing One, an excerpt from the program Init.c (available electronically; see "Resource Center," page 5). First, the struct is cleared and then the flag element is loaded with a bitwise-OR of a defined list of contents. The specific bit for each function is defined in the CAD-UL-supplied MON386 manual. Listing One shows the #defines for all the supported functions. For example, the OS_TASK_LIST bit in the flag is bit 0 and the OS_LOAD_INFO bit is defined as bit 11. For each bit set in the flag element, a corresponding function call must be supplied. Unused bits should be set to 0, and unused functions should have a null pointer, to allow an error message to be returned to the debugger if a mismatched monitor and debugger are invoked.

There are currently a total of 19 separate extension function types implemented in the debugger and monitor that are essentially similar in their implementation. Here, I'll use the OS_Task_List function as an example.

Listing Three (again from Init.c) shows the wrapper function used to perform near/far pointer conversion. This function, callOSTaskList, is the one loaded into the osCalls struct in lines 687-699 in Listing Three. It is used to build up a window of task states by repetitive calls. The function will return 0 when there are no more tasks in the list, otherwise it returns the ID number of the next available task. This method of returning both an element and a status byte that can be queried about any remaining elements is used throughout the implementation. This flexibility is used to allow for the retrieval and display of multiline OS Error Messages of an indeterminate length by recursively calling the same routine until all of the message is retrieved. If a task number of -1 is requested, then the function will return the total number of tasks in the OS.

Listing Three calls pOSTaskList, which is initialized to point to OSTaskList by the init_module routine found in the moninit.c file. init_module is called automatically on the monitor being loaded into the kernel and is also used to hook the monitor communication routines into the desired com-port interrupt handler. Listing Four shows the essential parts of the OSTaskList routine. The GetTask routine is called to obtain the specific information about the requested task. GetTask Call() converts the RT-task information into the format expected by CAD-UL's OS-aware function calls. While it has some specific RT pieces, it is similar to the Linux version in Listing Five. The true difference between the operating systems is evidenced by the GetTask routines. RTLinux information is read from the actual structures used in the RT kernel. Linux task or, more correctly, process information is read from the /proc filesystem directory entry for the process being debugged.

To summarize what happens when users open the Task List window in the debugger, this procedure is implemented:

1. XDB requests the number of tasks, by calling OS Extension Function #1 with a task number parameter of -1.

2. The monitor code performs a lookup to get the address of this function #1 from the osCalls struct. This points to the callOSTaskList() routine.

3. After performing the near/far conversion, callOSTaskList() then calls the function pointed to by pOSTaskList.

4. This was initialized on being loaded into the kernel to point to OSTasklList() and this routine is the first that looks at the data that was passed, for example -1.

5. The GetTaskCount() routine is called and the answer is passed back to the debugger after another near/far conversion is performed in the callOSTaskList() routine.

6. At this stage, XDB now has the number of tasks needed to fill the Task window; thus, it performs a call for each task, one by one, and is returned the task state, name, and ID number.

7. When the highest numbered task state is requested, the returned value indicates that there are no more and the completed window is displayed.

In this implementation for Linux, users then have the option of selecting any of the tasks in the Task window and requesting further information. There is always a trade-off in developing a remote debugger client between displaying all the information at once and the time needed to interrogate the OS and retrieve the data for the display. With a fast TCP/IP connection and relatively few tasks, this does not present a problem, but the same debugger may need to work over a much slower serial link. Not many embedded systems are running on a 650-Mhz Pentium III system, so uploading a large amount of data to the debugger may have a significant impact on the overall performance of the product. Quite often, an item is designed with minimum memory and minimum speed to keep the power demands as low as possible.

In contrast to the RTLinux implementation, the regular Linux monitor differs only in the final details of the GetTask() call. Instead of reading an internal structure that contains the task information as in RTLinux, the Linux /proc filesystem is read using standard operating system open() and read() calls. Listing Five is a section of the Linux file linuxos.c. It is easy to see the similarity between this file and the RTLinux version in Listing Four.

Implementing Extensions For a Proprietary or Custom OS

Whether you are using a large OS such as Linux -- as did our customer -- or a traditional RTOS, both dictate where to start adding in OS-aware features to your debugger. If you can modify something existing, then you can both reduce development time now and can lower subsequent maintenance costs each time a new release comes along.

Getting access to the OS developer or someone who is very familiar with the implementation is essential. Very few embedded operating systems are implemented as they are delivered. Commercial real-time operating systems are now available in source form, but even the traditional large vendors supply customization routines and multiple pieces that can be added or not, depending upon your needs.

To get started, obtain an evaluation copy of a commercial RTOS and the development tools and perform a thorough evaluation. Consider new visualization tools for gaining visibility into a running OS, using a completely separate task that logs data and presents it graphically. When compared to extending an existing monitor, this solution might more closely meet your needs because it presents a dynamic view of the data, rather than the snapshots available with a traditional debugger.

If you are already using a commercial RTOS, then some of the smaller vendors may be willing to trade OS licenses for the work you are doing, to allow them to pass on the benefits of an OS-aware debugger to present and future customers. Developing a good relationship with your OS company, in these days of consolidation, is recommended.

Conclusion

A complete set of code for the RTLinux Monitor for Beta 5 of RTLinux and the Linux User Monitor for the Version 2.2.11 kernel is available at the time of writing. Future versions will support RTLinux Version 2 and the Version 2.2.13 of the Linux kernel. This code and an evaluation version of the XDB debugger is available for download at http://www.cadul.com/ tools/ddj/. This version of the debugger is limited to a maximum of 10 source modules and 1000 symbols. A time-limited license that will remove this size restriction is available by sending an e-mail request to [email protected].

DDJ

Listing One

  ....
  memset(&osCalls, 0, sizeof(osCalls));
     osCalls.flag = OS_TASK_LIST | OS_TASK_INFO | OS_GET_TASK_ID |
             OS_LOAD_INFO | OS_QUERY | OS_COMMAND | OS_GET_ERROR_MESSAGE;
     osCalls.OS_Task_List = callOSTaskList;
     osCalls.OS_Task_Info = callOSTaskInfo;
     osCalls.OS_Get_Task_ID = callOSGetTaskID;
     osCalls.OS_Load_Info = callOSLoadInfo;
     osCalls.OS_Query = callOSQuery;
     osCalls.OS_Command = callOSCommand;
     osCalls.OS_Get_Error_Message = callOSGetErrorMessage;
  m386_setOScalls((OS_calls_t far *)&osCalls);
  .....

Back to Article

Listing Two

/* Function Pointers */
#pragma noalign  (s_calls)
typedef struct s_calls {
  OS_int flag;  /* flag for available functions */
#define OS_TASK_LIST     (1 <<  0)
#define OS_TASK_INFO     (1 <<  1)
#define OS_SET_TASK     (1 <<  2)
#define OS_GET_TASK     (1 <<  3)
#define OS_GET_TSS     (1 <<  4)
#define OS_SET_TSS     (1 <<  5)
#define OS_GET_TASK_ID     (1 <<  6)
#define OS_COMMAND     (1 <<  7)
#define OS_GET_OUTPUT     (1 <<  8)
#define OS_GET_ERROR_MESSAGE    (1 <<  9)
#define OS_LOAD      (1 << 10)
#define OS_LOAD_INFO     (1 << 11)
#define OS_SET_COMPLEX_BREAKPOINT_CONDITION  (1 << 12)
#define OS_GET_COMPLEX_BREAKPOINT_CONDITION  (1 << 13)
#define OS_DELETE_COMPLEX_BREAKPOINT_CONDITION (1 << 14)
#define OS_QUERY     (1 << 15)
#define OS_GET_FPU     (1 << 16)
#define OS_SET_FPU     (1 << 17)
#define OS_BREAK     (1 << 18)
/*  1 */OS_int far (*OS_Task_List)(OS_asbyte, OS_task_t far *);
/*  2 */OS_int far (*OS_Task_Info)(OS_int, OS_msg_t far *);
/*  3 */OS_int far (*OS_Set_Task)(OS_int, OS_int, OS_int far *);
/*  4 */OS_int far (*OS_Get_Task)(OS_int far *);
/*  5 */OS_int far (*OS_Get_TSS)(OS_int, OS_tss_t far *, OS_void far *);
/*  6 */OS_int far (*OS_Set_TSS)(OS_int, OS_tss_t far *);
/*  7 */OS_int far (*OS_Get_Task_ID)(OS_asbyte, OS_byte far *);
/*  8 */OS_int far (*OS_Command)(OS_asbyte, OS_byte far *, OS_msg_t far*);
/*  9 */OS_int far (*OS_Get_Output)(OS_asbyte, OS_asbyte, OS_msg_t far*);
/*  0 */OS_int far (*OS_Get_Error_Message)(OS_asbyte, OS_err_t far *);
/* 11 */OS_int far (*OS_Load)(OS_asbyte, OS_byte far *, OS_msg_t far *);
/* 12 */OS_int far (*OS_Load_Info) 
                    (OS_asbyte, OS_asbyte, OS_byte far *,OS_load_t far *);
/* 13 */OS_int far (*OS_Set_Complex_Breakpoint_Condition)
                    (OS_asbyte, OS_byte far *);
/* 14 */OS_int far (*OS_Get_Complex_Breakpoint_Condition)
                    (OS_asbyte, OS_msg_t far *);
/* 15 */OS_int far (*OS_Delete_Complex_Breakpoint_Condition)(OS_long);
/* 16 */OS_int far  (*OS_Query)(OS_asbyte, OS_byte far *, OS_msg_t far*);
/* 17 */OS_int far  (*OS_Get_FPU)(OS_int, OS_fpu_t far *);
/* 18 */OS_int far  (*OS_Set_FPU)(OS_int, OS_fpu_t far *);
/* 19 */OS_int far  (*OS_Break)(OS_int *);
#define OS_CALLS_MAX 19
} OS_calls_t;

Back to Article

Listing Three

/* ---------------------------------------------------------------------
 * MethodName: callOSTaskList
 * ---------------------------------------------------------------------

 * Description: wrapper function for near/far pointer conversion;
 * OSTaskList  can be found in file 'moninit.c'
 * ---------------------------------------------------------------------

 * Access: global
 * ---------------------------------------------------------------------

 * Parameter: length
 * -----------------------------------------
 * Parameter: pTask
 *            pointer to caller's task structure to be filled
 * ---------------------------------------------------------------------
 * Return: OS_SUCCESS, for successful result
 * ---------------------------------------------------------------------

 * Pre: none
 * ---------------------------------------------------------------------
 */
OS_int far callOSTaskList(OS_asbyte length, OS_task_t far* pTask)
{
 struct OS_near_task task;
 OS_int   result;
 task.ID = pTask->ID;
 task.state = pTask->state;
 task.length = pTask->length;
 task.name = (OS_byte*)pTask->name;

 result = (*pOSTaskList)(length, (OS_task_t*)&task);

 pTask->ID = task.ID;
 pTask->state = task.state;
 pTask->length = task.length;
 pTask->name = (OS_byte far*)task.name;

 return result;
}

Back to Article

Listing Four

OS_int OSTaskList(OS_asbyte n, OS_task_t* pTask)
{
 pthread_t p;
 ...
 p = GetTask(n);

 if (p)
 {
  static char szBuffer[20];
  sprintf(szBuffer, "%08x", p);
  pTask->ID = (OS_int)p;
  pTask->state = p->state;
  pTask->length = 9;
  pTask->name = szBuffer;
 }
 else
 {.....
      }
      return n == GetTaskCount() - 1 ? OS_NONE : n + 1;
}
pthread_t GetTask(int n)
{
 int  I = 0;
 pthread_t p = rtl_sched[i].rtl_tasks;
 for (i = 0; I <= n && p != 0; i++)
 {
  // first element is 0, second is 1, etc.
  if (i == n)
  {
   return p;
  }
  p = p->next;
 }
 return NULL;
}

Back to Article

Listing Five

OS_int OSTaskList(OS_asbyte n, OS_task_t* pTask)
{
 struct task* p;
 p = GetTask(n);
 if (!p)
 {
 pTask->ID = p->pid;
 pTask->state = p->state;
 pTask->length = strlen(p->cmdline) + 1;
 pTask->name = (void*)&(p->cmdline);

 if (p->pid == GetAttachedProcessID())
 {
  pTask->state = OS_DEBUG;
 }
 return n == GetTaskCount() - 1 ? OS_NONE : n + 1;
}

Back to Article