While developing and debugging, I regularly use Win32’s DebugBreak() and assertions to either break into or launch the registered debugger. Most of the time this works fine and accomplishes what I need. However, if the DebugBreak() call happens to lie within an exception handler, the call may be caught (and ignored) rather than launching the debugger as intended. To solve this problem, I devised a method which unconditionally breaks into the debugger.
About DebugBreak()
The primary purpose of DebugBreak() is to signal the debugger that an event has occurred. Internally, DebugBreak() raises the STATUS_BREAKPOINT (0x80000003) exception. The breakpoint exception is treated as any other: if unhandled, the application is terminated; however, it is typically (and often transparently) handled by the debugger.
Although all Windows platforms support DebugBreak(), under x86 builds it is advantageous to directly issue the breakpoint exception via the INT 3 opcode. The benefit is that the debugger will stop in application code rather than in system-level code within DebugBreak().
Additionally, certain Visual C++ library components internally use DebugBreak() (under x86 builds, MFC and the C runtime library use the INT 3 opcode rather than an actual DebugBreak() call, however the end result is the same), specifically assert() under the C runtime and ASSERT() under MFC. In debug builds, both of these functions eventually end up issuing a breakpoint exception if the assert condition fails.
Just-in-Time Debugging
Just-in-time (or JIT) debugging is an operating system feature that allows an external application the chance to gain control over an application that for some reason or another has generated an unhandled exception. Although the JIT settings typically refer to a debugger, it could be any application. For instance, Dr. Watson is the default JIT “debugger” under NT, but it gathers post-mortem information rather than operating as a true debugger.
Originally, just-in-time debugging was controlled via the “AeDebug” section in win.ini. With the introduction of NT and the subsequent move towards the centralized registry, these settings have migrated to:
HKEY_LOCAL_MACHINE \Software \Microsoft \Windows NT \CurrentVersion \AeDebug
Under Win95 and Win98 (both of which still support the win.ini settings), the registry setting takes precedence over any win.ini settings. Within the “AeDebug” key (or section) the “Debugger” value defines the application command line that should be executed if JIT is activated.
I have been unable to locate firm documentation on the actual supported parameter substitutions for the “Debugger” command line. However, after disassembly, reverse engineering, and empirical study, I have concluded that two command-line substitutions are required, and in the following order: the process id to be debugged, and an event to be signaled when the debugger has attached. Parameter substitution is performed using traditional printf()-style format specifiers. For example, Microsoft Visual C++ uses the following command line (see Figure 1):
C:\DevStudio\SharedIDE\BIN\msdev.exe -p %ld -e %ld
Basically, the JIT process is as follows:
1) An application causes an exception.
2) If the application does not handle the exception, the operating system steps in and traps the exception, suspending the application in the process.
3) The application error dialog is displayed (see Figure 2 and Figure 3). If a JIT debugger is registered, the user is offered the choice to debug the offending application. If not, the only option is to terminate the application. The “AeDebug” key also supports an “Auto” setting, that (if set to “1”) causes the registered JIT debugger to execute without user intervention via the Application Error dialog box.
4) Presuming the application is JIT-debugged, the operating system constructs the appropriate command line using the “Debugger” value from the “AeDebug” key and executes it.
5) The debugger attaches to the offending application via DebugActiveProcess() and signals the event specified in the command line, at which point the debugger has full control over the application.
Exception Handling
As previously mentioned, DebugBreak() results in a breakpoint exception which can be treated as any other exception using either operating system or language constructs. Both structured exception handling (__try/__except) and C++ exception handling (try/catch) allow an application to trap and optionally handle an exception. Unfortunately, these exception-handling mechanisms also trap (and optionally handle) debug breakpoint exceptions which may then interfere with the intended result of launching the debugger. Since failed assertions are generally used to signal exceptional conditions, it is often tedious to clutter standard exception handling clauses to special-case such breakpoint exceptions. Further, catch-all clauses (such as catch(...)) will transparently swallow breakpoint exceptions, never allowing just-in-time debugging a chance to execute.
Although exception handling is nothing new, it is becoming more and more prevalent in application code. Because of this and without careful consideration, DebugBreak() calls can often be transparently handled by such exception handling clauses, thus defeating the overriding purpose of DebugBreak(). For a number of reasons, it is not always practical or efficient to special-case breakpoint exceptions, therefore I came up with the solution presented below.
Forcing a DebugBreak()
What I needed was a method to guarantee that just-in-time debugging was activated for a DebugBreak() call. Although the actual process is relatively straightforward, a truly platform-independent implementation is clouded by the system differences between Win95/Win98 and NT.
Basically, there are two required steps to ensure that JIT is successfully activated during a call to DebugBreak(): if the current process is not running under the control of a debugger, launch the registered JIT debugger to attach and debug the process, then issue the DebugBreak() call. The debugger will then trap the call; execution will stop at that point, thus allowing further interactive debugging of the application.
In creating this solution, the first problem that I encountered was in determining if the currently executing process was running under the control of a debugger. Under NT and Win98, IsDebuggerPresent() returns TRUE if the current process is being debugged, FALSE otherwise. Alas, Win95 does not support IsDebuggerPresent(), and in fact it is not even included in the standard import libraries or as a Win95 stub export. Because of this, the function must be dynamically linked. Fortunately, Matt Pietrek in his excellent dissection of operating system internals [1] located an undocumented field in the process database that indicates whether or not the process is being debugged.
Using this information, my code first attempts to dynamically link to IsDebuggerPresent(). If that fails, then the undocumented hacks are performed on the process database to determine if the application is being debugged.
Following that, I attempt to locate the registered system JIT debugger. As previously detailed, I first search the registry and then the win.ini file to locate the debugger command line.
After successfully locating the registered debugger, it is a relatively simple task to create the signaling event and the required command line, and then launch the debugger. Of particular note, the created event handle that is passed to the debugger must be specified as inheritable, both in the CreateEvent() and CreateProcess() calls. After the appropriate process id and event handle substitutions are made, the resultant command line is passed to CreateProcess() to actually launch the debugger.
Once launched, it is necessary to wait for the debugger to assume control of the process before issuing the DebugBreak() call. This is done by waiting on the event handle that was initially passed to the debugger on the command line. All that remains once the event is signaled is to issue a breakpoint exception, at which point the process is suspended and the debugger gains control.
Source Code
fdbgbrk.cpp (Listing 1) contains the main source code for ForceDebugBreak(), which implements the techniques described in this article. ForceDebugBreak() will launch the debugger and cause a break, but only if the application is not currently being debugged. fdbgbrk.cpp also contains LocateAeDebugger(), which demonstrates how to locate the registered JIT debugger.
isdbgex.cpp (Listing 2) implements a function called IsDebuggerPresentEx(), which ForceDebugBreak() uses to determine whether or not the process is executing under the context of a debugger. In that same file, IsDebuggerPresent95() uses undocumented process fields and flags to determine whether a Win95 process is currently being debugged. I recommend Chapter 3 of Matt Pietrek’s book [1] for details on accessing these undocumented structures.
Finally, main.cpp (Listing 3) is a small application that tests ForceDebugBreak(). The program displays a message box and the user can press OK to call ForceDebugBreak(), or Cancel to attempt a DebugBreak() (see Figure 4). All of this takes place within a C++ try/catch clause. You can compile this test program under Visual C++ with this command line:
cl -GX -Zi fdbgbrk.cpp isdbgex.cpp main.cpp /Fefdbgbrk.exe
The source assumes that the application is running under Win95, Win98, or NT 4.0 or later. Further operating system version checks would be required if this cannot be guaranteed.
The code was developed with Visual C++ v5.0, compiles cleanly with STRICT defined and at warning level 4, and was tested on NT 4.0 (service packs 3 and 4), Win95, and Win98. It has not been tested on non-Intel builds or with other compilers.
Reference
[1] Pietrek, Matt. Windows 95 System Programming Secrets. Foster City, CA: IDG Books Worldwide, Inc., 1995.
Bret S. Pehrson started professionally programming educational software back in the days of Windows 3.0. Since then, he has worked on Multimedia authoring packages and 3-D rendering applications. He currently works as a professional consultant writing Windows CE applications.
Get Source Code