Phil Herold and Carla Merrill

Emulus is SAS Institute's enhanced 3270 emulator for the X Window System. Emulus started life as "phlem" (short for "Phil's Emulator"). When SAS Institute moved to networked Hewlett-Packard 9000 Series 700 workstations, phlem was used for 3270 emulation. Later, phlem was renamed "Emulus" and targeted as commercial software, but it did not have help available for the Help buttons in its dialog windows or for the Help item on its menu bar. Furthermore, SAS Institute didn't own a hypertext help system (compiler and viewer) that could support online-help development for Emulus. Helplus was created to fill this void and eventually became a separate application.

Clearly, building a powerful help system such as Helplus involves much more than we can adequately cover in a single article. Issues we faced included the user interface, file and tag structures, compiler and viewer design and implementation, server implementation, and interface design. In this article, we will focus on three areas: the help compiler, help viewer, and server program. We'll also touch on the design methodology that enabled us to achieve our goals in a timely manner.

Initially, we used Windows help as the model for the user interface and the structure of the help files for the compiler's input. We chose public-domain code for a hypertext widget as a starting point for the viewer.

Given our desire to emulate Windows help and the capabilities of the public-domain widget, we had to decide whether to assign a particular task to the compiler or viewer, the format of their input/output, and how particular tasks (such as searching) should be implemented. These decisions and their implementations influenced the evolution of the system into a separate software package.

The compiler was originally a separate C application; it became an X Window application when integrated into the same executable as the viewer even though it used only one X Window routine.

Using the input to Windows help as a model, we knew the compiler needed to process a help project file. This, in turn, required processing the RTF topic files it listed and producing a compiled help file in a format that would be read by the viewer and displayed. From the Windows help documentation, we learned about the content and structure of project files and RTF files. We decided to support the ROOT, TITLE, CONTENTS, and REPORT options. The [FILES] section of the project file, which lists the RTF files to compile, was supported in a format identical to Windows help.

Next, we decided which RTF tags the compiler would minimally need to support. The system would have to show text with highlighted links and allow users to select links to navigate through the help information. The public-domain hypertext widget supported both. We also wanted to emulate other Windows help-system features such as keyword search, history, bookmarks, and annotations. Therefore, the first tags we supported were the \footnote, topic, and pop-up link tags.

Next we defined the content and form of the compiler output, keeping in mind that the viewer needed to read the help file efficiently and make sense of the information. Designing the data for the compiler meant creating C-language data definitions (including C structures) that would map help-file topics and keyword information. For example, information for a particular topic (the topic context string, the topic title, the text for the topic, and the links within the text) had to be associated with that topic. In addition to data definitions, a mechanism for resolving links had to be provided. Early on, we assigned that task to the compiler, which affected its internal design and data structures. One effect was increased speed. In our design, the compiled link information is the text of the link, plus a number, which is an index into the list of topics that identifies the topic to be linked to. The compiled link information gives the viewer direct access to a topic, making the link-selection process fast.

Of course, the viewer could have accomplished this task quickly via a binary search on the topic context strings that are ordered alphabetically by the compiler. Then the compiler would not have had to resolve topic links. However, using the compiler to resolve links provided a second advantage: The compiler reported information about unresolved links that was extremely valuable while we were making changes and improvements to the interface. The frequent retagging and compiling required to verify and evaluate changes would not have been possible without the compiler's link-checking information. It was also easy to create a link map for maintaining help files later on.

We also knew that the viewer would have to perform a direct topic lookup in cases where Emulus would simply specify the help topic to display using the context string. This would occur if users selected the Help button on an Emulus dialog window. To make this process efficient at viewer time, we used a binary search of the context string information.

Another factor that affected the internal design of the compiler and its data structures was keyword management. The same keyword can be used for multiple topics, and topics typically have multiple keywords. This means that the compiler has to maintain keyword information independently of the help-topic information. Additionally, keywords have to be sorted to display alphabetically in the Search window, and duplicate keywords (the same keyword for multiple topics) have to be removed from the sorted list. To accomplish this, our compiler maintains a list of topics (again, using a number for topic indexes) for each keyword.

Once these processing mechanisms were designed, we designed the help-file format of the compiled output. UNIX standard I/O was the quickest format to implement. The Emulus code already contained routines that read a file into a buffer and present each line of text to a caller. The text could easily be parsed using the C sscanf function. Since the help viewer was initially a part of Emulus, which had these routines available and working, the format was attractive. The help-file format was essentially text, so we decided that writing the help file to stdout would be sufficient and, perhaps, useful. Precedent certainly exists for this approach in other UNIX applications. In any case, the output could always be directed from the UNIX prompt to a real file on disk. Writing to stdout also eliminated the need for a compiler option to specify the destination of the compiler output.

With the design in place, we started writing and testing code, building parts of the system from the bottom up. First came the code that would read a help project file and compile the list of RTF files. Once written, a full-screen debugger was used to test the code, verifying the internal data. Then we wrote the code that would read each RTF topic file and parse it. This code would look for the RTF tags we needed to support and compile the topic information.

Topic links were resolved in a traditional second pass of the compiler after all of the topic information was read from all of the RTF files listed in the project file. This pass through the data also processed keyword information (associated topics were resolved to topic indexes). The last code developed was a simple function that wrote the help file to stdout. With this function in place, verification of the output simply consisted of eyeballing the data spit out by the compiler in the terminal window.

We started by making the help viewer an integrated part of Emulus. The original public-domain hypertext widget defined functions to create a hypertext widget, set the text in a hypertext widget (SetText), and retrieve the text from a hypertext widget (GetText). To load the hypertext widget with text, our help application called the SetText routine, passing it the text. Within the text, tag characters instructed the hypertext widget to display an item as a link. Link text was displayed in a different color and underlined. The color was controllable through an X resource. The tag characters were also controllable through X resources, although they defaulted to braces ({}), which conformed to our help application.

We added more link types (for example, pop-up link text that used a dotted underline and pop-up link text that had normal text color and no underline). To accommodate these new links, more tag characters were added to the text to tell the hypertext widget how to visualize the link. (This approach is cumbersome, however, so the next release of Helplus uses link segments.)

We added two more parameters to the SetText routine:

• The text for the title was set with the addition of the titleFont resource. However, this was changed again when we put the title outside of the scrolled-window hypertext widget and into a hypertext widget of its own, which was not part of the scrollable area. This allowed the topic title to remain at the top of the window even if the user had to scroll to see additional topic text. To accomplish this, a different hypertext widget was passed with the title parameter set to a value. SetText was then called with the rest of the text for the scrollable hypertext, with a NULL value for the title.
• We did titles completely differently from Windows help, partly because we were not yet familiar enough with Windows help syntax. Our implementation of the viewer always used the topic-title text from the ${\footnote statement as the first line of the topic. We added the titleFont resource to distinguish the font of the topic-title text from the rest of the topic text. Of course, we realized later that Windows does not do this. The text that is part of the ${\footnote statement is used for the History, Search, and Bookmark dialog windows. To display a topic title as part of the topic text, the Windows help-file author must also type the title (and any appropriate font change) in the topic text.
• An annotate parameter was added, indicating if the help topic had associated annotation text (the hypertext widget itself was not bothered with the details of the annotation). This parameter was simply a Boolean value. If True, the hypertext widget displayed a paper clip in front of the title.

GetText retrieves the text from the hypertext widget, usually with the special tag characters removed. Our help application used GetText to get the text for the Copy function. GetText was eventually changed to accommodate the different types of links and to remove picture tags from the returned text.

The original public-domain code also displayed the hypertext widget, presumably parented to a scrolled-window widget. Action (translation) routines were already defined to track the mouse-pointer movement and calculate when the mouse was over link text. In this case, the appearance of the mouse pointer changed to a hand. If the user selected link text, the hypertext widget would call a callback function (defined by the hypertext widget's owner) and pass it the link text and the length of the link text. The caller was then responsible for determining what to do with the link, presumably reloading the hypertext widget with other text.

The hypertext widget had to be modified to return more information in the callback structure. These modifications implemented features in the interface that made our hypertext linking more flexible and extendible. For example, our first pop-up topics were spring-loaded, appearing only as long as the user held down the mouse button. This process was handled entirely by the hypertext widget. However, we wanted to put both topic and pop-up links within a popup. This required the popup to stay on screen after the user released the mouse button and to remain until it was clicked on. Enabling this behavior required making the callback structure reflect the behavior to the hypertext widget's owner--our help application. The information had to include whether or not users had clicked to remove the popup or selected a link within the popup.

We also had to modify the callback structure to support the Annotate feature. If the annotation paper clip is selected, the callback structure needs to indicate this and allow the owner to display the annotation dialog. All of this additional information was specified through separate reason codes, which are a part of every X Window callback structure.

Six weeks into our project, the help system had bitmap (bitonal) support; a highly readable, attractive, proportional font; and menu and button features in place. We achieved these milestones because of the design methodology we used, although we assessed our results differently. The "writer" judged the application in terms of its ability to present information efficiently and attractively. The "developer" judged it by its functionality and performance. Despite our different perspectives, we both realized that our help system was developing an identity of its own. The idea of making it a distinct application appealed to both of us and seemed the logical next step.

Making the help system a separate software package would insulate Emulus from the newly developed help-system code, which greatly increased the executable size of Emulus and made it more volatile late in its development cycle. However, making the help system an independent application was a problem.

Even after we decided to externalize the help viewer from Emulus, running the help system as a server was not an obvious solution. Our original inclination was to exorcise the viewer code from Emulus and make it a stand-alone program for viewing help files. Then users would get a new copy of the viewer program each time it was invoked (either through an application or from the UNIX shell prompt).

A colleague suggested using a server program that runs in the background, waiting for other processes (clients) to ask for service. In the case of Helplus, the client could be an application or Helplus itself (if it were invoked from the UNIX prompt when it was already up and running).

A server implementation would insulate Emulus from the help system's code, and it provides additional benefits:

• It allows sharing of resources (help-file data and picture data) when multiple instances of Helplus are invoked. Help-file data is the memory devoted internally to data structures that map the help-file information. In Helplus, this can be considerable, since the entire help file is brought into memory and left there. (In the next release of Helplus, the help file is kept on disk and only brought in one page at a time as the user selects topics.)
• A Helplus client can start up faster, since the server is already running and in memory. Additionally, if the help file is already being viewed, it is not reread from disk.
• A server allows changes in bookmarks or annotations to be easily broadcast to other instances of the same help-file view. For example, if a user is viewing the same topic in multiple Helplus viewer windows and adds or deletes an annotation or bookmark, all of the Helplus viewer windows are updated simultaneously.

1. The application obtains the value of this special property set by the help server. This value is the window ID created by XCreateSimpleWindow when the server is started. (This window is never realized by the help server.)
2. With XChangeProperty, the application can change two properties on this window: the help filename to display and the topic name within that help file.
3. With XSendEvent, the application sends the help server a message. The data in this message consists of a window ID for the client, a generic help handle, and the interned atoms for the two properties (help filename and topic name). For the first request by an application to the server, the help handle passed is NULL.
4. The server sends a message back to the client indicating the success or failure of the request. The indication is the existence of a message, which can be displayed by the client. The returned message also contains the value of the help handle. This value is always saved by the client and used in subsequent requests to the help server.

The key factor in our design methodology was that the writer actually used the interface while the developer was developing it. The writer's motivation for frequent compilation and viewing was to edit small amounts of text; online editing is more reliable when done in small increments. The writer could also verify the appearance of the text frequently. Changes could be made to existing text and incorporated into any new text immediately. As the amount of compiled help information grew, more attractive and efficient ways of presenting it became clear through repeated viewing and use of the compiled help file. If this editing phase had been postponed, improvements to the interface would have been less apparent and there would have been little time to make and test improvements.

Another factor in our methodology was quick response time. The writer compiled and displayed the current help file an average of three times per day, allowing new elements to be tested as soon they were available. It also revealed bugs early on. The developer's fast response to fixing bugs and making other changes was critical to the evolution of the help system.

Our daily cycle consisted of these steps:

1. The developer made the latest code (compiler and viewer) available to the writer.
2. The writer compiled, edited, and displayed the most-recent version of the help file. Editing included checking format, grammar/spelling, style, and links.
3. The writer told the developer about any bugs in compilation and display of the information.
4. Any deficiencies or enhancements to the interface were discussed, based on the most recent display.
5. The developer changed the code to fix bugs and implement enhancements, including the support of new tags.
6. The writer used the new tags for new text and retagged old text to take advantage of any new enhancements.

Helplus has become a strategic internal tool that serves as the native help system for the SAS System running on UNIX hosts. The most recent version includes support of a richer RTF tag set, Windows help macros, and text flowing. In addition, help files created for Windows can now be compiled by Helplus. This allows online help development for the SAS System to proceed across UNIX and Windows with a single set of RTF files.

Copyright © 1995, Dr. Dobb's Journal