10 Steps to Embedded Dialog Management

When a modal dialog is used, the Windows Dialog Manager provides the keyboard navigation facilities that we are all used to — tab navigation, cursor keys, mnemonics, button activation. Creating a modal dialog by calling the DialogBox()/DialogBoxParam() function(s) creates a modal loop and uses a dialog box procedure built into Windows itself to provide all that functionality.

Embedded dialogs are those where a parent window or dialog contains not only child control windows, but also child dialogs. They can make it easier to have a very sophisticated and configurable user interface. An obvious example is the Windows NT/2000/XP Task Manager.

Unfortunately, when dealing with embedded dialogs, the support provided by the dialog manager falls short. In particular, an inner dialog does not have its keyboard processing carried out correctly by the dialog manager, and exhibits behavior such as button flashing and focus that is "trapped" inside the inner dialog and prevented from passing back into the outer dialog. This can lead to a very frustrating experience for those users who prefer to use keyboard navigation over the mouse. (The sample applications for this article provide an example of such frustrations: Run the step2 example, move to the Edit2 control with the mouse, and then try to quit the application from the keyboard without pressing Alt-F4.)

Building on the functions described in a number of previously published Tech Tips [1, 2, 3, 4], this article will highlight the issues involved in providing fully embedded dialog keyboard functionality in 10 steps, culminating in a dynamic, embedded dialog application. For brevity, only the salient code changes are shown in this article, but the full code for all steps is supplied in the archive, available online. It is progressed incrementally, so you may use a differencing tool (perhaps WinDiff or your source-code control system) in order to follow precisely the evolution of the code through each step. Each step creates a console window and traces out the various transition and control notifications, so that you can easily monitor the progression of the dialog behavior as the functionality builds.

By highlighting the issues involved in custom keyboard navigation, demonstrating some of the techniques for its implementation, and providing functions that can be taken and applied in any Windows programs, this article may assist developers in providing more sophisticated, high-quality user interfaces.

Onedlg

The dialog that we will emulate for the first nine steps is shown in Figure 1. Its behavior is as one would expect from the layout, specifically that Radio1, Radio2, and Radio3 form a radio group, and that the tab order is Radio1-Radio2-Radio3-Check2-Button2-Edit2-OK-Cancel. While the dialog is pretty simple, it nevertheless contains sufficient complexity to exercise the common embedded dialog challenges.

In order to demonstrate these issues, the dialog is synthesized at run time from two separate dialogs. The parent, or outer, dialog (Figure 2) consists of the radio group, and the OK and Cancel buttons. The embedded, or inner, dialog (Figure 3) consists of the checkbox, push-button, and static/edit pair.

The outer dialog has the same style (DS_MODALFRAME | WS_POPUP | WS_CAPTION | WS_SYSMENU) as the original dialog, and the inner dialog now has the WS_CHILD style.

Step 1 — No Custom Keyboard Handling; Using DialogBox()

In this step, the outer dialog (Figure 2) is created in the same way as the single dialog (Figure 1) in the original application. In its handling of WM_INITDIALOG, the outer dialog creates an instance of the inner, using CreateDialog() [5], and passing the outer dialog as the parent. It also passes the InnerDialogProc() as the dialog procedure, which is nothing more than a stub in the first few steps. (Note that the inner dialog could be successfully created with NULL as the dialog procedure, although this results in a totally nonfunctional child [5].)

When you run this program, you will find that most keyboard aspects that involve the inner dialog are nonfunctional. You cannot use the TAB or cursor keys to move within the inner dialog, or between inner and outer. You cannot select or activate controls within the inner dialog.

We can summarize the problems thus:

1. No navigation into, or out of, the inner dialog.
2. No navigation within the inner dialog.
3. Mnemonics do not work for the inner dialog controls when in the outer dialog.
4. Mnemonics do not work for the inner dialog, even when in the inner dialog.

This is due to the Windows dialog manager — accessed via the DialogBox() family of functions — not providing support for embedded dialogs. It fails to process the TAB key with respect to the inner dialog along with its peer child windows, and misses it out of both forwards and backwards navigation. Furthermore, the processing of mnemonics only works for top-level children.

Step 2 — No Custom Keyboard Handling; Using CreateDialog()

Problems 1 and 3 will be dealt with in the subsequent steps described in this article. Two and 4 can, however, be easily implemented using the Win32 IsDialogMessage() [5] function.

BOOL IsDialogMessage(HWND hDlg, MSG *pmsg);

This function determines whether the given message is for the given window and, if so, processes the message by dispatching to the window, as well as carrying out any other necessary tasks in the dialog (such as changing states of affected radio and push buttons). IsDialogMessage() is primarily intended for use with modeless dialogs, but can also be used for embedded dialogs.

The single call to DialogBox() in step1.cpp is changed in step2.cpp (see Listing 1). This change results in problems 2 and 4 being corrected. Once the focus is within the inner dialog, mnemonics and the TAB and cursor key presses work fine. But the transitions between the inner and outer dialogs still do not work, nor do mnemonics for the outer dialog from within the inner dialog, and vice versa.

Importantly, the handling of the Return and Escape keys is no longer performed when the focus is in the inner dialog, meaning that the user cannot dismiss in the affirmative (OK, via the Return key) or negative (Cancel, via the Escape key), thereby adding a fifth problem. Furthermore, the OK button continues to bear the default button highlighting even when the Button2 control has the focus: problem 6.

Step 3 — Handling TAB Key Presses

The answer to the lack of keyboard movement and mnemonic response between the outer and inner dialogs starts with getting into the window message-processing loop. As you may know, all Windows applications have within them — whether in the application main()/WinMain() functions, or hidden inside the Windows Dialog Manager — a message loop, of the general form shown in Listing 2.

This loop takes one message at a time from the application (or rather thread) message queue, and dispatches it to the window to which it was sent or posted. (The TranslateMessage() function is used to convert certain raw keyboard messages into character messages [5].)

One of the reasons that these three functions are separate is so that developers may intercept messages before they are dispatched, in order to customize application behavior. It is this ability that we will make use of for most of the customizing steps in this article. The usual form for customized message loops is of the form shown in Listing 3.

In step 3, the handling of TAB key presses is modified in order to provide expected behavior. As we saw in step 2, intradialog control transitions work for both inner and outer dialogs. It is the interdialog transitions that are missing. There are four transitions, Radio1/2/3 => Check2, Check2 => Radio1/2/3, Edit2 => OK, and OK => Edit2. In order to provide these changes, we examine the message retrieved by GetMessage() to see whether it is "special." For example, if msg.message == WM_KEYDOWN, then it is a key press. If the virtual key code in msg.wParam is VK_TAB, then the TAB key has been pressed. We then look at the currently focused window, which is the one the user is "leaving." Finally, we get the state of the Shift key, via the Win32 function GetKeyState(). Putting these together, we can capture the attempt to TAB forward from Edit2, and rather than allowing the default behavior of the dialog manager to move the focus to Check2, we move the focus to the OK button. Listing 4 shows the relevant code for this and the other three custom transitions.

Step 4 — Navigating Around Radio Buttons

While the TAB key presses are largely handled by the previous step, when one tabs or, as in this case, back-tabs (by pressing Shift-TAB) into a radio group, a question is raised as to which control to set the focus to. Obviously in this case, we know that the prior control to Inner:Check2 is Outer:Radio1-3, and that we could manually check which radio button, if any, is selected and set the focus to it. But in many real-world embedded dialog applications, the controls on either side of the dialog junctions may be dynamically configured and, hence, unknown (at least at compile time). Thus, just setting the focus to the prior/posterior control can lead to incorrect behavior, as you see in step 3 (Figure 4).

This issue is dealt with by use of the GotoDlgCtrlMaybeRadio() (see the sidebar "Custom Keyboard Navigation Functions"). By replacing the four calls to SetFocus() (see Listing 4) with GotoDlgCtrlMaybeRadio(), we get the correct behavior, as can be seen in Figure 5.

Step 5 — Handling Cursor Key Presses

If you run onedlg.exe and step4.exe, the only difference in the cursor key handling is that when you cursor back — by pressing either the left (VK_LEFT) or up (VK_UP) arrows — then step4.exe takes you to Check2, whereas onedlg.exe demonstrates the correct behavior in taking you to Edit2. The reason the former happens is that the dialog manager moves the focus back to the previous control, which is the inner dialog, and sets the focus to it. The inner dialog then sets focus to its first member, Check2.

The fix for this step is straightforward. The conditional in Listing 5 is inserted after the test for TAB key presses in the message loop, resulting in correct behavior (Figure 6).

I should point out that though only a single, manually handled transition was required in this application, it is possible to have very complex control key behavior (see the sidebar "Cursor-Key Navigation"). The good news is that the cursor keys are very rarely used to navigate other than for radio groups, so you can probably ignore the issue otherwise.

Step 6 — Default Buttons: Button Status Highlight

When the focus is on a push-button, pressing the Return key will activate that button, and it will send a message (WM_COMMAND with BN_CLICKED code) to its parent as if it was clicked by the mouse. (The same thing happens when the space bar is used.) Dialogs have the concept of a default push-button, which is activated in response to a Return key press when no push-button is focused. This could lead to some confusion in that a user may not readily appreciate which button would be activated. To resolve this, Windows provides special highlighting to denote which push-button will be the default. When the focus is with a control other than a push-button, the dialog's default button will have a thicker border, as can be seen in Figure 6. When the focus moves to another push-button, it gets the thicker border, indicating it will get the Return-activation, and the default button is redrawn as a normal button to indicate that it will not. Once focus shifts to a nonpush-button, the default button gets its thicker border again.

The implementation so far does not deal with this issue, resulting in the erroneous dialog shown in Figure 7. However, with a little coding (Listing 6), this can be rectified.

When Button2 gets the focus, it switches off any default nature of the OK and Cancel buttons (see Figure 8) with UndefButton() (see the sidebar "Custom Keyboard Navigation Functions"). Note that UndefButton() does not remove the default characteristic from the OK key, merely the thicker highlighting, but that is fine because the Button2 push-button will "eat" the key-press, so it will not get to the outer dialog.

When Button2 loses the focus (to any window), the outer dialog is instructed (via the DM_SETDEFID message) to set the OK button as its default. For this to work, the Button2 control must have the BS_NOTIFY style; otherwise, it will not send the BN_SET/KILLFOCUS notifications to its parent, the inner dialog.

These requirements for button notification and the forwarding of notifications in the embedded dialog's window procedure place quite a requirement on the inner dialog's implementation. While the amount of code is not onerous, this technique does couple the two implementations, making dynamic creation and activation of inner dialogs along the lines of ActiveX controls [6] more challenging. We will see some simple dynamic embedded dialog techniques in step 10.

Step 7 — Default Buttons: Return & Escape Key Handling

In all the previous steps, when the focus is on a control within the inner dialog, pressing the Return and Escape buttons has no effect. The expected behavior is that when the user presses the Escape key, this is equivalent to pressing the Cancel button (if any) on the dialog. The dialog manager sends a WM_COMMAND with id == IDCANCEL to the dialog procedure.

When the user presses the Return key, one of two things happens. If the control with the focus is a push-button, then it is clicked. Otherwise, the dialog default button, if any, is clicked.

To solve this, the IDOK and IDCANCEL command notifications received in the inner dialog procedure need to be sent on to the outer dialog (see Listing 7).

Step 8 — Custom Keyboard Handling for Mnemonics: Moving to the Controls

Mnemonics provides the ability to navigate to and activate controls by name (see the sidebar "Mnemonics"). This step will deal with the first part of this facility, navigation. Using the GetChildFromMnemonic() and GotoDlgCtrlMaybeRadio() functions (see the sidebar "Custom Keyboard Navigation Functions") we insert the code in Listing 8 into the message loop.

The code recognizes WM_SYSCHAR — the message sent when a mnemonic is pressed — and then determines whether it has been pressed within the inner or the outer dialog. Because the mnemonics within each dialog work fine, we only need to step in when the mnemonic is pressed for one while the focus is within the other. The GetChildFromMnemonic() function is called to locate the control within the requisite dialog. If a match is found, then GotoDlgCtrlMaybeRadio() is used to move the focus, and bHandled is set to TRUE to prevent the call to DispatchMessage().

Step 9 — Custom Keyboard Handling for Mnemonics: Activating the Buttons

The code introduced in step 8 answers the mnemonic navigation, but as you will see when you run it, it does not handle the activation.

Before one of the radio group's button is selected (meaning that none are checked — see Figure 9), navigating out of the inner dialog into the radio group as a result of a mnemonic (e.g. pressing Alt-R) causes the focus to move to the requisite radio button, but does not select that button. This is not the natural behavior of dialogs.

When moving from the outer dialog to the inner dialog's Check2 control — which is an autocheckbox (BS_AUTOCHECKBOX) — it is not checked/unchecked, again contrary to the behavior of dialogs. Conversely, activating Button2 by mnemonics moves the focus to it. When push-buttons are activated by mnemonics, they never get the focus but are just clicked (except when two or more share the same mnemonic character, in which case the next one in the dialog order from the focused control is clicked).

The answer to all of these issues is the GotoDlgCtrlAndClick() function (see the sidebar "Custom Keyboard Navigation Functions"), which moves the focus to the given control, and causes it to be clicked if it is (1) a button; (2) not sharing a mnemonic with another control in the dialog; and (3) of the appropriate button type (checkboxes and radio-buttons must be of the autostyle: BS_AUTOCHECKBOX and BS_AUTORADIOBUTTON, respectively). Replacing the GotoDlgCtrlMaybeRadio() calls, inserted in step 8 with GotoDlgCtrlAndClick(), results in the mnemonic behavior one would expect (see Figure 10).

Step 10 — Dynamic Switching Between Embedded Dialogs

So now that we've seen some of the techniques, let's apply them to a more generic and realistic example. The application is expanded in step 10 to incorporate three inner dialogs, each one being hidden/shown in response to the selection of one of the radio group button selections. The application implementation is very similar to that of step 9, with the difference including the addition of three window handle arrays that hold handles to the inner dialogs, their first controls, and their last controls. These handles are used in the established navigation logic to work with the currently selected and displayed dialog. Listing 9 shows the WM_INITDIALOG handler of the outer dialog, incorporating the main differences with step 9.

By leveraging the techniques described in this article, this application works correctly in respect to TAB and cursor key presses, as well as mnemonics. Figures 11 and 12 show the other two "faces" of the application. In each case, the only change is that the visible inner dialog has been activated and the others deactivated by using the Win32 ShowWindow() function [5].

Summary

You should now be better armed to take on embedded dialogs, and making them seamless, enhancing your user's experience and your product's quality. There is one small caveat. While the functions described in this article and the supporting tips have evolved over a number of versions of Windows, from 3.0 onward, I have not had the opportunity to check the behaviors of the current implementations with every operating-system flavor, but they have been tested on Windows 98, NT4, 2000, and XP.

References

[1], [2] Wilson, Matthew. "Custom Keyboard Navigation with Radio Buttons, Parts 1 & 2," Windows Developer Magazine, May and November 2002, respectively.

[3], [4] Wilson, Matthew. "Custom Keyboard Navigation with Mnemonics, Parts 1 & 2," Windows Developer Magazine, November and December 2002, respectively.

[5] http://msdn.microsoft.com/.

[6] Brockshmidt, Kraig. Inside OLE. Microsoft Press, 1995.