Channels ▼
RSS

Tools

Postmortem Debugging

Source Code Accompanies This Article. Download It Now.


Saving the Stack to the Log File

The task of writing all return addresses from the stack to the log file is done by the function LogStackFrames (Listing Four). Each stack frame's return address is prefixed with the tag "$ADDRESS:". MapAddr.exe just looks for these entries and interprets them as program addresses.

Listing Four: LogStackFrames.

void   LogStackFrames(PVOID FaultAddress, char *eNextBP)
{      char *p, *pBP;                                     
       unsigned i, x, BpPassed;
       static int  CurrentlyInTheStackDump = 0;
   ...
       BpPassed = (eNextBP != NULL);
       if(! eNextBP)
       {
          _asm mov     eNextBP, eBp   
       }
       else 
           fprintf(sgLogFile, "\n  Fault Occurred At $ADDRESS:%08LX\n", 
                                   (int)FaultAddress);
                             // prevent infinite loops
       for(i = 0; eNextBP && i < 100; i++)
       {      
           pBP = eNextBP;           // keep current BasePointer
           eNextBP = *(char **)pBP; // dereference next BP 
           p = pBP + 8; 
           // Write 20 Bytes of potential arguments
           fprintf(sgLogFile, "         with ");        
           for(x = 0; p < eNextBP && x < 20; p++, x++)
               fprintf(sgLogFile, "%02X ", *(unsigned char *)p);
           fprintf(sgLogFile, "\n\n");          
                  if(i == 1 && ! BpPassed) 
               fprintf(sgLogFile, "*************************************\n"
                                 "         Fault Occurred Here:\n");
        // Write the backjump address
        fprintf(sgLogFile, "*** %2d called from $ADDRESS:%08LX\n", 
                                  i, *(char **)(pBP + 4));
        if(*(char **)(pBP + 4) == NULL)
            break; 
       }
    ...

}

LogStackFrames also prints the first 20 bytes of that area of the stack where arguments to the functions are stored. The determination of whether these are really arguments or local variables would need further inspection by the assembler code—or a look at the source code. According to the language specification, C and C++ functions put their arguments on the stack from right to left, whereby the address following the return address always holds the first argument of the function, if any. Interpreting these values requires a little bit of thinking, but usually this "manual" approach is sufficient.

Although Windows offers a function StackWalk(...) to walk the stack, for simplicity, this is done using simple pointer dereferencing. If LogStackFrames is called from an assert (where BP is unknown), one line of inline x86 assembler is used to get the value of the BP register.

MapAddr.exe

The program MapAddr.exe is called from the command line with two arguments—a log file path and a map file path. It first reads the map file and builds a sorted list of all addresses in it. It then creates a copy of the log file (with the suffix ".out") where, for each stack address found by the tag "$ADDRESS:", it inserts a line with the appropriate function name. It also prints the offset of the return address from the function start (extracting the actual line within the function from the address would be a more difficult task—although not impossible). The result is the log file including function names, as in Figure 1. The function names are mangled with their arguments, as all sources for this article are C++. Therefore, the function

int FunctionC(int x, int y) 

becomes

?FunctionC@@YAHHH@Z 

in the map file of Visual C++ 6.0. If you use the Visual C++ source browser, you can use the goto definition command to jump to the function body while the cursor is positioned on this mangled name.

Analyzing Client Program Faults

By adding one function call to an application, you can now analyze a problem postmortem that occurred on any machine anywhere else. You now know in which function the bug occurred and from where it was called. This certainly is an improvement, but it may only be one piece of an overall strategy for analyzing program faults on client machines. There are a number of considerations:

  • Asserts. To find hidden bugs early, use as many asserts as possible and leave them in the code in the shipping release versions. In the release version, an assert should enable postmortem debugging by calling the function LogStackFrames with both parameters NULL (which will determine BP by an assembler call). Clearly, an assert should always give a message as loudly as it can. "Loud" in this context may mean a modal dialog—it can also mean creating an e-mail or any other kind of signal in the case of a server application.
  • Sometimes asserts that fail may not be errors at all, leading to bug reports that are created solely by an assert. However, that problem is outweighed by the advantage of finding serious bugs quickly and early. Some people argue that there are performance reasons for leaving asserts out in release versions. In most cases, this argument is moot. Performance problems almost always have only two solutions—a profiler and/or better design.
  • Log files. Finding all the ways to the problem source often requires further information. Therefore, an application should always create a log file on startup and fill it with information about a program's basic conditions: the program and exact OS version, amount of memory, loaded documents, and so on. During execution, all commands should be logged together with their parameters. When a fault occurs, the current state (free memory, global application states, and the like) is of interest together with the stack frames. All this will lead to more success in locating the problem source.
  • Although writing log files is good in general, it can sometimes create problems. I saw a server application writing gigabytes of log files over the course of a few days until the disk was full. Log file writing requires control mechanisms (such as ring logging) that prevent such problems. Besides that, it's always helpful if the number of log statements written can be modified by users.
  • Automation. Using MapAddr.exe in a professional context, of course, implies that all map files of all versions must be archived and accessible for postmortem problem analysis. Also, in a professional context, you may write batch scripts (or extend MapAddr.exe), so that the program version is extracted from the log file and the appropriate map file is selected automatically.

Conclusion

No application is error free. The postmortem analysis approach I present here helps you identify and resolve hidden and unreproduceable errors with little impact on the program itself. The technique also reduces the problem of sometimes unclear and guess-based user reports. Using postmortem debugging together with logging and asserts, you can clearly see what happened.


Stefan is a software developer in Berlin specializing in cross-platform development, user interfaces, and reengineering. He can be contacted at StefanWoe@gmail.com.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video