Channels ▼
RSS

.NET

Post-Mortem Debugging Revisited


Several years ago I wrote an article entitled Post-Mortem Debugging that included source code to create stack dumps with function names from C/C++ programs that crashed on client sites without any modification on the client's machine. This was done in two steps (available on Windows and Linux/UNIX):

  1. Install a fault handler in your application that dumps the call stack to a text file if an error has occurred.
  2. Run a program (MapAddr.exe) to add function names to the memory addresses of the stack dump by matching them to a map file from the program's build.

Since then, I've received feedback that led me to a number of different approaches which I summarize in this article.

Retrieving Line Numbers from the Stack Dump (Windows Only)

Visual Studio stores all debug information in a program database which resides in a file with the extension ".pdb". This database file is created for debug builds by default, but it is easy to create them for optimized release builds as well:

In the Project Setup dialog select the following 2 Options:

  • In the C/C++ Section
    Debug Information Format: Program Database (/Zi)
  • In the linker Section
    Generate Debug Info. YES

This will generate a .pdb file for the selected build, which should be archived for each released version. It also enables the possibility to debug the release version of your program just as you would do with the debug version.

.pdb files are stored in a binary format that is not documented, but it's easy to access it through a COM API. The Debug Information API: DIA is provided by Microsoft starting from Visual Studio 2008. Using this API, it is straightforward to get the following information for every code address:

  • Unmangled function name (human readable including the list of parameters)
  • Source file name
  • Line number within that source file

Figure 1 shows a stack dump created from the same FaultApp as used for demonstration in Post-Mortem Debugging. While adding the line number to the stack dump may only seem like a "nice to have" feature, it ultimately proved to be very valuable once I started using it. Before having line numbers I always had to examine the entire function. Now I only have to check a single line of code, and the problem often becomes obvious immediately.


****************************************************
*** A Programm Fault occured:
*** Error code C0000005: ACCESS VIOLATION
****************************************************
***   Address: 004398E8
***     Flags: 00000000
****************************************************
*** CallStack:
****************************************************

  Fault Occured At $ADDRESS:004398E8
                   ======== 000398E8 == belongs to FunctionC(int,int)
    File d:\projects\mapaddr\faultapp.cpp Line 29
                            with 64 00 00 00 10 00 00 00 80 FE 12 00 00 00 00 00 00 E0 FD 7E 

***  0 called from $ADDRESS:00439969
                   ======== 00039969 == belongs to FunctionB(int)
    File d:\projects\mapaddr\faultapp.cpp Line 41
                            with 64 00 00 00 54 FF 12 00 00 00 00 00 00 E0 FD 7E CC CC CC CC 

***  1 called from $ADDRESS:004399E5
                   ======== 000399E5 == belongs to FunctionA()
    File d:\projects\mapaddr\faultapp.cpp Line 51
                            with 00 00 00 00 00 00 00 00 00 E0 FD 7E CC CC CC CC CC CC CC CC 

***  2 called from $ADDRESS:00439A68
                   ======== 00039A68 == belongs to main
    File d:\projects\mapaddr\faultapp.cpp Line 64
                            with 01 00 00 00 70 34 4D 00 B8 34 4D 00 84 5C FF C1 00 00 00 00 

***  3 called from $ADDRESS:0044B573
                   ======== 0004B573 == belongs to __tmainCRTStartup
    File f:\dd\vctools\crt_bld\self_x86\crt\src\crt0.c Line 327
                            with 

***  4 called from $ADDRESS:0044B32D
                   ======== 0004B32D == belongs to mainCRTStartup
    File f:\dd\vctools\crt_bld\self_x86\crt\src\crt0.c Line 195
                            with 00 00 00 00 00 00 00 00 00 E0 FD 7E 05 00 00 C0 C8 FF 12 00 

***  5 called from $ADDRESS:7D4E7D42
         with 

***  6 called from $ADDRESS:00000000
************************************************************

Figure 1

Accessing the pdb file involves a number of COM-Calls bundled together in a class file. Adding the usage of this class to MapAddr was simple and straightforward. MapAddr can now be called with pdb files as well (as before) using map files. The full source code can be downloaded here.

64-Bit Windows

The fault handler of the first version used x86 assembler calls to unwind the stack. But when porting to 64 bits, I found no way to write stack frames reliable with assembler instructions with optimization enabled. While there are a number of Win32 API functions to unwind the stack, only one (RtlCaptureStackBackTrace) seems to work on x64 as expected. It can only retrieve 62 stack frames which is sufficient in most cases, although this limit seems questionable.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video