Other Post-Mortem Techniques: Windows Mini Dumps

Minidumps store the current state of a program in a binary file that can be loaded into Visual Studio or WinDbg. When loaded into a debugger, a minidump shows the full symbolized stack and all local variables just as it would during a debug session. That is, of course, more comfortable than symbolizing a logfile.

Minidumps can be created a number of ways:

• Using Dr.Watson to be called on a program fault.
• Calling it explicitly from within the program itself ; i.e., by a installed Structured Exception handler, as in the approach described in the first issue of this article
• Using a tool such as WinDbg or MiniDumpTool

The disadvantages of minidumps are that they are Windows specific and they cannot be batch processed (as possible using a utility such as MapAdrr.exe).

Other Post-Mortem Techniques: Windows Error Reporting

You probably have seen a dialog similar to Figure 2, which was introduced with Windows XP:

[Click image to view at full size]

Figure 2

To tell the truth, I was surprised to learn that Microsoft will send error reports even for programs that I have written but never published. However, I realized that I could use this mechanism for my personal applications. Windows Error Reporting (WER) is a service offered by Microsoft basically for free. To sign up you need a VeriSign certificate. With the certificate you can sign up for WER and obtain the error reports that users have accepted to upload using the dialog box above. These reports are just minidumps as described above.

Some considerations about WER:

• Applications, that have installed a exception handler using SetUnhandledExceptionFilter (as used in the first article) sometimes crash without calling the installed exception filter. In such cases, WER still can be used, thus giving a additional last chance to find bugs.
• WER does not offer the possibility to add any additional data or description by users. When using a own mechanism for error reporting (possibly a small separate utility) it may ask users to report information about how to reproduce the problem as well as sending additional data files to reproduce the bug. Such a utility also can be called when other problems occur, that do not lead to a crash.

It seems best to use WER as an additional service to report problems. Note also that there has been a number of reports about problems signing up to WER, especially when using a certificate from a different provider than VeriSign.

Other Post-Mortem Techniques: Google Breakpad

According to Google, breakpad "is a library and tool suite that allows you to distribute an application to users with compiler-provided debugging information removed, record crashes in compact "minidump" files, send them back to your server, and produce C and C++ stack traces from these minidumps. Breakpad can also write minidumps on request for programs that have not crashed. Breakpad is currently in use by Google Chrome, Firefox, Google Picasa, Camino, Google Earth, and other projects as well."

This is basically the same mechanism as propagated by MapAddr in my first article, but of course google has enhanced the functionality:

• Breakpad works on multiple platforms (Windows, Linux, Mac). The minidump format has been adapted on other platforms.
• It includes functions to transmit a crash report directly to your http server
• It can also handles other errors such as C++ pure virtual function calls or invalid parameter calls and it also can be triggered explicitly to create a crash report.

Breakpad seems to be well developed and can probably handle a number of conflicts that a simple utility like MapAddr cannot (i.e., loading DLLs to a different base address than the preferred one). Unfortunately its documentation is only fragmented. Even the documentation about installing and compiling is not correct. To compile breakpad for Windows, you first must install Python, then run the script:

src\tools\gyp\gyp.bat src\client\windows\breakpad_client.gyp

While the current directory is the root directory of breakpad. This will create Visual Studio Solution and project files; for other platforms, make files are provided.

Summary

There are a number of ways exist to capture stack dumps from program crashes and add symbol information to them. Such information is very helpful to resolve bugs and to improve the quality of a program. Post-mortem debugging saves time in finding bugs, as a stack dump often is sufficient to reproduce and solve a bug. The effort needed to acquire this information is minimal.