Bob as a Macro Processor Library
Turning a tiny OO language into a macro language
Brett Dutton
Brett recently completed his studies at Queensland University of Technology in Brisbane, Australia. He now heads RAX Information Systems, a consulting firm specializing in application software, and can be contacted at [email protected].
How many applications have you written that have never needed modifications because of user demands, suggestions, or customization? Not many, I'll bet. Luckily, application macro languages give you and users a way of modifying and customizing the application without recompiling.
Macro languages are user-programmable interfaces to applications. They allow users (and developers) to access internal functionality and to customize or extend the application. Macro languages are used in many applications, including emacs, AutoCAD, Brief, WordPerfect, Excel, and Lotus 1-2-3, to name a few.
Besides adding the ability to customize, macro languages also provide benefits such as easy prototyping, automated testing, distribution of multiple configurations, smaller executables, better application design, and easier upgrades.
I recently added a macro language to a Syntax Directed Text Editor (STex) as part of a project at Queensland University of Technology (QUT) in Brisbane, Australia. Since STex will be used by first-year undergraduates, many factors influenced the choice of macro language: ease of programming, ease of implementation, and consistency with the programming models supported by university teaching methods.
Among the macro languages we considered were:
- XLisp. Originally written by David Betz and available on most archive sites, XLisp made the list because it has a proven track record with AutoCAD. However, Lisp is considered a little difficult for first-year undergraduates to understand.
- ReXX. The scripting language for VM/CMS, OS/2, and Amiga, ReXX has implementations available on most platforms, and again, a proven track record. However, the project coordinator did not want to use ReXX.
- Bob. Also written by David Betz and presented in Dr. Dobb's Journal ("Your Own Tiny Object-Oriented Language," DDJ, September 1991), Bob is a readily available language with a C/C++ syntax implemented in ANSI C. The modularity concepts native to C/C++ made Bob particularly desirable to the university. That the language is object oriented was also a bonus.
Changes to Bob
Bob was originally designed by David as a stand-alone interpreter. In 1994, it was extended into a language for building online conferencing systems (see "An Online Conferencing System Construction Kit," by David Betz in Dr. Dobb's Information Highway Sourcebook, Winter 1994). Furthermore, David has since written a version of Bob that runs as a Windows DLL; see the accompanying text box entitled, "Callable Bob." By adding an API to the original Bob, my extensions take a different course. However, to make my implementation of Bob complete, I had to modify the original code. This article focuses on the changes I made to the stand-alone Bob interpreter to turn it into a macro processor library, and on how you can use that library. The complete source code to the Bob macro processor library is available electronically; see "Availability," page 3. For details on the Bob language and syntax, refer to David's 1991 article.
Originally, Bob was only meant to be run once. One of my first changes was to make Bob reentrant. The ability to initialize on load-up lets you load up another file, which in turn will run another initialization, and so on. The position on the stack is now maintained by the call-return set of functions rather than set up on the first call.
In addition, I provided remote function access by creating the new data type DT_RCODE for remote functions. Data values of this type are handled in the same way as strings, but are used to interface with application functions. The remote functions are available to Bob after they have been registered with the Bob API.
Next I made the Bob macro processor its own process so that it works independently of the application. Communication with the interface is by socket communication. It uses two sockets: one error socket that interrupts processing; and a general socket for normal communication.
To extend internal functionality, I extended the number of internal functions to include a useful subset of C functions; see Table 1.
I also added the ability to initialize on load-up by using a function of the same name as the file being loaded up. Finally, I maintained backward compatibility so that saying "make Bob" will build an executable that works as originally designed.
Figure 1 represents a model of how the Bob macro processor library works. Although complicated, it is not necessary for you to fully understand it unless you would like to modify the Bob library. Luckily, my API hides all the complexity within a few functions.
Bob API
Listing One is a sample application that demonstrates use of the Bob API. (The API itself is fully documented in the bob.h source file.) Although this particular example was written with the X Toolkit Intrinsics (Xt), you do not need an X Window-based system to use it. All of Bob's processing is done through the function BobCheckClient, which polls the Bob communications socket for traffic and processes any packets that have come through. It returns True if there is a packet to process, and False if not. In the Xt library, the function XtAppAddInput "listens" on a socket. Bob has a similar function, BobBlockWait, that can be passed a socket to listen on. BobBlockWait returns when traffic arrives on either the Bob comms socket or the passed socket.
The power of Bob is in the external functions defined by the application. This application defines two functions (message and error), but unlimited external functions can be added. External function arguments are limited to three data types: number, string, and NIL. This limitation arises because the data-packet has only been made to transfer these data types across the socket.
Because Bob is a typeless language, a variable is defined as a VALUE. It is up to the function to test the type of the values being passed to it and either coerce the value or reject it as an error.
In examining Listing One, note that:
- int BobInitialize (void) initializes the Bob macro processor and returns either the socket on which to listen or --1 on failure.
- int BobLoadFile (char *fname) sends a request for Bob to load and compile a file. The filename is passed in fname and needs neither a suffix of .bob nor a path if it is in either the home directory or the path pointed to by the environment variable BOBLIB. Errors in the compilation are reported to stderr, and the processing stops. If the file loads successfully, Bob looks for the function of the same name as the file (illegal characters are replaced by underscores). If this function exists, it is executed and deleted upon completion. This function is considered the initialization function.
- int BobAddFunction (char *fname,BobExternFunc func) adds an external function to Bob. When that function is called by Bob or the application, the function will execute. The definition of BobExternFunc is typedef int (*BobExternFunc)(int, VALUE *args, VALUE *etval);.
- int BobExecute (char *name,int cnt,_) executes the passed function name, passing the arguments to the function. The argument list is in pairs, indicating the type of the argument and the value. The exception to this case is when NIL is passed. It has type only, no value. An example of running the print function with a number of arguments would be BobExecute ("print", 4, DT_ STRING, "Hi", DT_INTEGER, 20, DT_NIL, DT_STRING, " World\n" ); and the result would be Hi 20nil World.
- char *BobGetString (char *buf, int len, VALUE *val) is a utility function to extract a string from a value. It is used in remote functions.
- int BobGetInteger(VALUE *val) is a utility function to extract the integer part of a VALUE.
- int BobCheckClient(void) polls the socket, checking if there is anything to read, and processes one waiting packet. If there are more packets, they will be ignored until the next call to BobCheckClient.
- int BobExtArgs(int argc, int mn, int mx,VALUE *retval) checks the number of passed arguments. mn is the minimum number of arguments that this function needs, while mx is the maximum. If there is no minimum number of arguments for the function, set mn to --1. If there is no maximum, set mx to --1. If mn or mx is set to --1, the external function must verify argument counts.
- int BobExtCheckType(VALUE *arg, int type,VALUE*retval) checks the type of the passed VALUE and handles or checks the type of arguments.
- int BobReturnValue(VALUE *retval, int type,_) returns a value to the function that invoked the Bob function call. The use of this function is similar to BobExecute.
- int BobTalkTerm(void) terminates the Bob macro processor. It's sufficient to simply exit, but using this function is a little cleaner.
- int BobBreak(char *fmt,_) sends a break signal to Bob. If Bob encounters a break, it stops any processing and returns to the EventLoop. This function is useful to break large, slow functions. Its arguments are the same as those for sprintf-type functions.
- void BobBlockWait(int fd) waits for activity on either the Bob sockets or on the passed socket and returns when activity is detected. None of the packets are processed. This function was originally used when the application was written using Xlib, and it was difficult to add a socket to the application. It was easier to have Bob keep an eye on the X socket and return if there was activity on the passed socket or on the Bob socket.
When you start the example application, it will give you a list of functions to execute and files to load. Try these by typing them into the appropriate dialog box, then selecting OK (but don't press Return; the application isn't that smart).
Limitations and Future Enhancements
Bob is not without its limitations. For instance, it currently has limited memory. The memory size is defined by the macro SMAX (Stack Max Size) defined in bob.h but it could be changed to use virtual memory. Memory is currently statically allocated because of garbage collection requirements.
Bob currently does not support floating-point arithmetic, although this could be implemented via classes. The internal functions are only a subset of C-type functions. It is possible to extend the number of internal functions, but it would be better if the Bob macro processor library stayed free from application-dependent code. The file bobfcn.c contains all the internal functions; see Table 1. It may also be necessary in the future to extend the data types that external functions can handle.
You can now link Bob into your applications and get all the benefits that a macro language provides.
Callable Bob
David Betz
David is a DDJ contributing editor and can be contacted through the DDJ offices.
Last fall I presented a version of my Bob programming language that was extended to support an object store designed for use as the basis for building an online conferencing system (see "An Online Conferencing System Construction Kit," Dr. Dobb's Information Highway Sourcebook, Winter 1994). While that Bob interpreter was easy to extend with additional built-in functions, it was essentially a stand-alone interpreter. Bob was the main program, and extensions were called as subroutines. This works well in many applications but falls short when trying to use Bob as an extension language for an existing application.
To solve this problem, I recently designed a version of Bob that runs as a Windows DLL. Along the way, I separated Bob into several modules so that each could be used independently. The memory manager is now an independent module that can be used as the basis for other languages that need a heap with automatic garbage collection. I've also separated the Bob interpreter from the Bob compiler, since some applications only need to run already-compiled code. In fact, I've separated the run-time library from the rest of the interpreter so it is possible to run programs that only need the intrinsic functions without any library at all. To make things simpler, I've included all of these modules in the Bob DLL even though they are logically separate. The complete source code to this version of Bob is available electronically; see "Availability," page 3.
Windows DLLs have only a single data segment, even though several applications may be linked to them at the same time. This was a problem, since Bob had many global variables, most having to do with the memory manager. My first step in turning Bob into a callable library was to move all of the globals into context structures and add a parameter to every function to explicitly pass in the appropriate context. I created two context structures: the interpreter context, which contains the bytecode interpreter variables as well as the memory manager variables; and the compiler context, which contains the compiler and scanner variables. The compiler context also points to an interpreter context so that the compiler has access to the memory manager for creating objects.
Passing the compiler and interpreter contexts into each function explicitly makes it possible to create more than one context at a time. This allows a multithreaded program to have multiple threads, all executing Bob programs independently. It also means several programs linked to the same Bob DLL can operate without interfering with each other.
Now I'll show how to invoke the Bob DLL to create a simple read/eval/print loop for Bob expressions. First, it is necessary to create an interpreter context, as in Example 1(a). The first parameter is the size of the heap, and the second is the size of the stack. The second line sets up an error handler. Bob will call this error handler whenever an error occurs passing it an error code and relevant data.
If you need access to the run-time library functions, that is arranged by Example 1(b). The second line sets up a handler that the interpreter will call to get the address of a function handler, given a function name. This is necessary when the interpreter restores a saved workspace because the saved workspace format on disk contains only the names of library functions, not their addresses. This allows saved workspaces to work correctly even after the DLL has been rebuilt, causing the function-handler addresses to change.
It's now necessary to initialize the compiler, as in Example 1(c). This creates a compiler context with the specified interpreter context. The numeric parameters are the sizes of the compiler-bytecode staging buffer and the literal staging buffer.
The Bob memory manager is a compacting, stop-and-copy garbage collector and can change the address of objects when a garbage collection occurs. Because of this, the memory manager must know about all variables that could contain a pointer to an object in the heap. The variables that the interpreter uses are contained within the interpreter context structure and can therefore be located by the memory manager. However, it is sometimes useful for a memory-manager client to maintain its own pointers into the heap. The Bob memory manager allows for this by providing the function ProtectPointer to register an object pointer with the memory manager. This registers the specified pointer with the memory manager and guarantees that its value is updated whenever the garbage collector moves the object it points to.
This leaves the read/eval/print loop itself; see Example 2. Bob does all of its I/O through "streams." A stream is an object with some data and a pointer to a dispatch table. The dispatch table has pointers to handlers to carry out various stream operations. At the moment, there are handlers for getting and putting characters and a handler for closing the stream. The call to CreateStringStream creates a stream that allows the Bob compiler to read characters from the string. The interpreter context structure contains pointers to the standard I/O streams that must be set up by the client of the Bob DLL. These streams should arrange for characters to be read and written to the standard input and output of the application.
The call to CompileExpr compiles a single Bob expression and returns a compiled function which, when called with no arguments, will cause the expression to be evaluated.
CallFunction calls a function with arguments; see Example 3(a). The arguments after argumentCount are passed to the specified Bob function. They are of type ObjectPtr (a pointer to a Bob heap object), and argumentCount indicates their number. You can also call a Bob function by name, using Example 3(b).
Of course, the Bob DLL has many other functions. It contains a full compliment of object-creation and access functions for creating and manipulating objects of type ObjectPtr, as well as functions to control the interpreter.
This is just my first step in making Bob easier to embed in applications. I plan to extend the Bob language to support full function closures and optional arguments. I'll also add a "fast load" format for storing precompiled Bob code in disk files. This would make it possible to distribute Bob functions without including the source code, a necessary feature for using Bob to build commercial applications.
Example 1: (a) Bob interpreter context; (b) accessing the run-time library functions; (c) initializing the compiler.
(a) InterpreterContext *ic = NewInterpreterContext(16384,1024);
ic->errorHandler = ErrorHandler;
(b) EnterLibrarySymbols(ic);
ic->findFunctionHandler = FindLibraryFunctionHandler;
(c) CompilerContext *c = InitCompiler(ic,4096,256);
(d) ObjectPtr val;
ProtectPointer(ic,&val);
Example 2: The read/eval/print loop.
for (;;) {
printf("\nExpr> ");
if (gets(lineBuffer)) {
Stream *s = CreateStringStream(lineBuffer, strlen(lineBuffer));
if (s) {
val = CompileExpr(c,s);
val = CallFunction(ic,val,0);
printf("Value: ");
PrintValue(ic,val,ic->standardOutput);
CloseStream(s) ;
}
}
else
break;
}
Example 3: Calling a Bob function (a) by reference; (b) by name.
(a) ObjectPtr CallFunction(InterpreterContext ic, ObjectPtr function, int argumentCount,...); (b) ObjectPtr CallFunctionByName(InterpreterContext ic, char *functionName, int argumentCount,...);
Figure 1 How the Bob macro library works. Figure 2 Sample dialog box.
Table 1: Bob internal functions.
<b>Function Description </b>
char chr (ascii_value); Converts the ASCII value into a string.
string date_time (); Returns the current date and time in
the format "Mon Nov 21 11:31:54 1983"
string downcase (string); Converts the passed string to lowercase.
string editcase (string); Converts the passed string to edit case.
Edit case is where the first character
after a space is uppercase and the rest
are lowercase.
val exec_function (fname[,arg1 Executes the passed function name with
the arguments. The number of arguments
must be consistent with the function
that is being called. This function returns
the value that the function would have
returned.
int fclose (file); Closes the passed file.
file fopen (fname,mode); Opens the passed file in the mode: r
for read, w for write.
int gc (); Does a garbage collection.
int getc (file); Returns the next character from the file.
string getenv (envname); Returns the string associated with passed
environment variable name.
bool keyboard_quit (); Stops the current processing. Returns NIL.
list list_functions (); Returns a list (vector) of function names.
bool load_bob (filename); Loads the Bob macro file. If the file
is not available, then returns NIL.
The file is not compiled into memory
until the current function is processed.
string newstring (size); Returns a blank string of the passed size.
vector newvector (size); Returns a vector of the passed size.
bool nothing (); This function does nothing. It could be
used for disabling key translations.
int print (val1 [,val2 Prints the passed values to stdout.
[,...valn]]);
int putc (file,char); Puts the passed character to the file.
int sizeof (value); Returns the number of elements in a
vector or the length of a string,
or 1 for any other type of value.
int str_to_nam (string); Returns the passed string as a number.
int strchr (string,char); Returns the position of char in
string. If returns < 0, then
the string was not found.
int strlen (string); Returns the length of the passed
string. This is an alias for sizeof.
int strstr (string1,string2); Returns the position of string2
in string1. If < 0 is returned, then
the string was not found.
string substring Returns the substring starting at
(string,start-pos,[len]); the position pos for the length len.
If the length arg is not there, then
returns the rest of the string. Pos of
0 is the beginning of the string.
int system (command); Sends a command to the operating
system. Returns the OS exit code.
string typeof (value); Returns the type of the passed
string, which is one of the following:
NIL, CLASS, OBJECT, VECTOR, INTEGER,
STRING, BYTECODE, CODE, DICTIONARY, VAR, FILE.
string upcase (string); Converts the passed string to all uppercase.
string val_to_string (value); Converts any value to the equivalent string
and returns it.
string version (); Returns the current version string of Bob.
Listing One
/* example.c: Exemplifies the use of the Bob macro processor library
* Copyright (c) 1994 Brett Dutton
*** Revision History: * 13-Dec-1994 Dutton Initial coding
*/
/* Description: */
/* includes */
#include <stdio.h>
#include <stdlib.h>
#include "bob/bob.h"
#include <X11/Intrinsic.h>
#include <X11/StringDefs.h>
#include <X11/Xaw/Label.h>
#include <X11/Xaw/Command.h>
#include <X11/Xaw/Box.h>
#include <X11/Xaw/Dialog.h>
/* macros */
#define APPNAME "example"
#define VERSION APPNAME " 1.0 By Brett Dutton"
/* typedefs */
/* prototypes */
void AppCheckBob ( XtPointer cl_data, int *fid, XtInputId *id );
void Quit ( Widget w, XtPointer cl_data, XtPointer call_data );
void Break ( Widget w, XtPointer cl_data, XtPointer call_data );
void LoadOk ( Widget w, XtPointer cl_data, XtPointer call_data );
void ExecuteOk ( Widget w, XtPointer cl_data, XtPointer call_data );
int Message ( int argc, VALUE *arg, VALUE *retval );
int Error ( int argc, VALUE *arg, VALUE *retval );
void ShowError ( char *msg );
/* variables */
/* these resources are usually external */
static String resources[] = {
"*example.width: 300",
"*example.height: 400",
"*quit*label: Quit",
"*break*label: Break",
"*Command*background: green",
"*message*label: Message:",
"*message*width: 275",
"*error*label: Error:",
"*error*width: 275",
"*value: ",
"*load*label: Load file:",
"*load*loadok*label: Ok",
"*load*width: 275",
"*execute*label: Execute function:",
"*execute*executeok*label: Ok",
"*execute*width: 275",
};
/* global widgets */
Widget message, errordia, load, loadok, executedia, executeok;
/* functions */
* Function: main -- main function
* Returns: Nothing
*/
void main ( int argc, char *argv[] )
{
XtAppContext app_context;
Widget topLevel;
Widget box, quit, brkwid;
int bobSock;
/* create a application */
topLevel = XtVaAppInitialize ( &app_context, APPNAME, NULL, 0,
&argc, argv, resources, NULL );
/* create all the buttons and dialogs for the application */
box = XtVaCreateManagedWidget ( "box", boxWidgetClass, topLevel, NULL );
quit = XtVaCreateManagedWidget ( "quit", commandWidgetClass, box, NULL );
brkwid = XtVaCreateManagedWidget ("break", commandWidgetClass, box, NULL);
message = XtVaCreateManagedWidget ( "message",dialogWidgetClass,box,NULL );
errordia = XtVaCreateManagedWidget ("error", dialogWidgetClass,box, NULL );
load = XtVaCreateManagedWidget ( "load", dialogWidgetClass, box, NULL );
loadok = XtVaCreateManagedWidget ( "loadok",commandWidgetClass,load,NULL );
executedia = XtVaCreateManagedWidget ( "execute", dialogWidgetClass, box,
NULL );
executeok = XtVaCreateManagedWidget ( "executeok", commandWidgetClass,
executedia, NULL );
/* set up all the callbacks for the buttons */
XtAddCallback ( quit, XtNcallback, Quit, 0 );
XtAddCallback ( brkwid, XtNcallback, Break, 0 );
XtAddCallback ( loadok, XtNcallback, LoadOk, 0 );
XtAddCallback ( executeok, XtNcallback, ExecuteOk, 0 );
/* initialize the bob interface language */
if ( ( bobSock = BobInitialize ( ) ) < 0 ) {
fprintf (stderr,"Unable to initialize Bob\n" );
exit(1);
}
/* add this socket to the event loop for monitoring */
XtAppAddInput ( app_context, bobSock, (XtPointer)XtInputReadMask,
AppCheckBob, NULL );
/* register the external functions with BOB */
BobAddFunction ( "message", Message );
BobAddFunction ( "error", Error );
/* load up the application defaults macros */
BobLoadFile ( "." APPNAME "rc" ); /* user definitions */
/* this has just been put in to demonstrate calling BOB functions */
BobExecute ( "print", 4, DT_STRING, "Hi ", DT_INTEGER, 20,
DT_NIL, DT_STRING, " World\n" );
/* create windows for widgets and map them */
XtRealizeWidget ( topLevel );
/* loop for events */
XtAppMainLoop ( app_context );
}
* Function: AppCheckBob -- The is the work proc called when there is * input from Bob
* Returns: Nothing
*/
void AppCheckBob ( XtPointer cl_data, int *fid, XtInputId *id )
{
/* Call bob to get the events of the Bob comms socket */
BobCheckClient ( );
}
* Function: Quit -- Exits from the windows system
* Returns: Nothing
*/
void Quit ( Widget w, XtPointer cl_data, XtPointer call_data )
{
BobTalkTerm (); /* shutdown comms with Bob */
exit ( 0 );
}
* Function: Break -- Exits from the windows system
* Returns: Nothing
*/
void Break ( Widget w, XtPointer cl_data, XtPointer call_data )
{
BobBreak ( "BOB Inturrupted" );
}
* Function: LoadOk -- The load dialog is complete and to load up file
* Returns: Nothing
*/
void LoadOk ( Widget w, XtPointer cl_data, XtPointer call_data )
{
String str; /* filename to load */
char msg[500]; /* Error message */
Arg xargs[1]; /* New value */
/* get the string and try to load it */
str = XawDialogGetValueString ( load );
if ( BobLoadFile ( str ) ) {
/* clear the box if no error */
XtSetArg ( xargs[0], XtNvalue, (XtArgVal)"" );
XtSetValues ( load, xargs, 1 );
} else {
/* send an error to the error box */
sprintf ( msg, "Unable to load file: %s", str );
ShowError ( msg );
}
}
* Function: ExecuteOk -- Execute dialog is complete
* Returns: Nothing
*/
void ExecuteOk ( Widget w, XtPointer cl_data, XtPointer call_data )
{
String str; /* function to execute */
char msg[500]; /* error message */
Arg xargs[1]; /* New value */
/* get the string and try to load it */
str = XawDialogGetValueString ( executedia );
if ( BobExecute ( str, 0 ) ) {
/* clear the box if no error */
XtSetArg ( xargs[0], XtNvalue, (XtArgVal)"" );
XtSetValues ( executedia, xargs, 1 );
} else {
/* send an error to the error box */
sprintf ( msg, "Unable to execute function: %s", str );
ShowError ( msg );
}
}
* Function: Message -- Displays a message in the dialog boc
* Returns: Tells Bob if it is an error or not
*/
int Message ( int argc, VALUE *arg, VALUE *retval )
{
char msg[500]; /* message to put in dialog */
Arg xargs[1]; /* New value */
/* make sure that there is 1 args */
/* make sure that it is a string */
if ( ! BobExtArgs ( argc, 1, 1, retval ) ) return ( FALSE );
if ( ! BobExtCheckType ( &arg[0], DT_STRING, retval ) ) return ( FALSE );
BobGetString ( msg, sizeof ( msg ), &arg[0] );
XtSetArg ( xargs[0], XtNvalue, (XtArgVal)msg );
XtSetValues ( message, xargs, 1 );
return ( BobReturnValue ( retval, DT_INTEGER, TRUE ) );
}
* Function: Error -- Displays a error in the dialog boc
* Returns: Tells Bob is there is an error or not
*/
int Error ( int argc, VALUE *arg, VALUE *retval )
{
char msg[500]; /* error to put in dialog */
Arg xargs[1]; /* New value */
/* make sure that there is 1 args */
/* make sure that it is a string */
if ( ! BobExtArgs ( argc, 1, 1, retval ) ) return ( FALSE );
if ( ! BobExtCheckType ( &arg[0], DT_STRING, retval ) ) return ( FALSE );
BobGetString ( msg, sizeof ( msg ), &arg[0] );
ShowError ( msg );
return ( BobReturnValue ( retval, DT_INTEGER, TRUE ) );
}
* Function: ShowError -- Displays the passed message in the error dialog box
* Description:
* Returns: Nothing
*/
void ShowError ( char *msg )
{
Arg xargs[1]; /* New value */
XtSetArg ( xargs[0], XtNvalue, (XtArgVal)msg );
XtSetValues ( errordia, xargs, 1 );
}
Copyright © 1995, Dr. Dobb's Journal
