Bluetooth & Remote Device User Interfaces

A protocol for building UIs for small devices

Richard is a development engineer at FlexiPanel Ltd and coauthor of Data Mining and Business Intelligence: A Guide to Productivity. He may be contacted at rhoptroffflexipanel.com.

I originally developed the FlexiPanel protocol in 2002 as an IrDA infrared protocol for engine tuning in hobbyist racing cars. The engine was controlled by a microcontroller that needed to be fine-tuned with a number of preset values. Incorporating an onboard user interface (such as an LCD display and keyboard) to enter the tuning parameters was impractical—it would be too expensive, heavy, bulky, and fragile. At the time, PDAs with infrared capability were becoming popular, so it seemed logical to use one for the user interface. The FlexiPanel protocol was thus conceived as a way to let embedded systems create UIs on any suitable device that users might have at hand.

However, the protocol's limitations, and its wider potential, soon became apparent. IrDA might be wireless but, in practice, the communicating devices must be held steady within a one-foot (30 cm) range of each other and within line of sight. On the other hand, emerging radio technologies, such as Wi-Fi, Bluetooth, and ZigBee (a low-power protocol), offered 330-foot (100 m) range. Nor did they need exterior real estate—a tremendous product design advantage in terms of cost, aesthetics, and reliability in hostile environments.

Consequently, we decided to migrate the technology to Bluetooth, on the grounds that it was widely implemented on remote devices and suited the ad hoc nature of the connection between appliances and remote handhelds. In addition, Intel had just committed incorporating Bluetooth into its Centrino 2 chipset. While Wi-Fi and ZigBee remain viable transport layers, they currently are not implemented widely enough on handheld devices.

At the same time, it became clear that the FlexiPanel protocol's potential went far beyond engine tuning. It provides a Human Machine Interface for smart dust (intelligent devices comprised of a microcontroller and a few external components), thus far extending their range of application. It offers a UI for "headless" (no screen or keyboard) single-board computers. This promises cost reductions for point-of-sale systems, computer-operated machine tools, and the like, while freeing operators from control panels. In traditional Windows applications, the protocol offers the possibility of a supplementary, roaming UI. Though rarely vital to Windows applications, roaming capability is a liberating addition to any application that might have anything to say to users while away from their desk.

In short, the FlexiPanel Bluetooth Protocol is a remote user-interface service for computers, electrical appliances, and other machinery. A FlexiPanel server resides on the application and holds a UI database that reflects the appliance's human-machine interface needs. A FlexiPanel client can connect at any time, read the database, and display the UI. Users may then control the application from the client device. Using Bluetooth, clients can be up to 330 feet away without need for line-of-sight communication. FlexiPanel clients have been implemented on a range of PDAs and cellphones and are freely available (http://www.flexipanel.com/DownloadsIndex.htm).

Like many higher level protocols such as OBEX file exchange, FlexiPanel sits on top of the RFCOMM serial port emulation layer of the Bluetooth protocol stack (Figure 1). It is not part of the official Bluetooth Standard. However, the Standard is relatively open in that anyone is free to create FlexiPanel clients, and FlexiPanel server licenses are royalty free (http://www.flexipanel.com/RemoteControlsAPI/index.htm).

As with all emerging technologies, early Bluetooth devices suffered from clunky operation and compatibility problems. This was particularly true of service discovery—finding out what Bluetooth devices are in range and the services they provide. Many 2002-era devices were limited to preconnection over a virtual serial port, rather than providing service discovery APIs. In the last year, Microsoft's support for Bluetooth in Windows Sockets has made service discovery a one-click process that even Amazon boss Jeff Bezos would be proud of.

A User Interface Service

From the application's perspective, FlexiPanel is a GUI service just like any other. It can create controls and subsequently change the properties of those controls. If users modify a control, the application is sent a notification message. The application doesn't need to know that the UI is displayed remotely. Conversely, just like a web browser, the FlexiPanel client is generic and does not need to know anything about the server it is providing a UI for. (Client software is generally free, too.) Table 1 lists the types of control that may be created. Under the hood, the FlexiPanel protocol is based on just 12 basic types of message passed between client and server (Table 2).

The main differences between the FlexiPanel protocol and regular UI services are:

• The nature of the client's UI may be unknown. The controls displayed will always be logically correct but appearances may vary between different client devices. Compare the same slideshow controller UI on a PDA (Figure 2) and a cellphone (Figure 3). If the client device can be anticipated in advance, certain additional preferences can be requested, such as a particular control layout (Figure 2) or keyboard accelerators (Figure 3).
• The connection might be broken at any time, for example, if the client's batteries fail or the client goes out of range. The appliance must enter a failsafe state if connection is lost at a critical moment.
• FlexiPanel servers might be small, low-cost devices, such as Parallax's FlexiPanel peripheral for BASIC Stamps (http://www.parallax.com/), based on a low-end 8-bit microcontroller (Figure 4). Consequently, system requirements on the server side must be extremely lean and communication succinct (unlike XML). The remote client device takes over as many responsibilities as possible. For example, a server is not required to buffer any I/O, manipulate any floating-point numbers, or make any conversions between single-byte characters and Unicode.
• Bluetooth is a multipoint communication protocol. It is possible for a FlexiPanel server to manage UIs on up to seven remote devices at once. This is only implemented on high-end systems such as Windows servers, but is useful for applications such as restaurant table service, where several wait-staff can connect to the same server at once. A quick disconnect mode is also available, where servers close the connection immediately after connection has been established and control panel information sent. This lets servers send UIs to a large number of clients connecting asynchronously, although the clients have no chance to send any messages back to the server.
• The FlexiPanel protocol plays no role in authentication, encryption, error detection, or power management. These are expected to be managed by other layers of the Bluetooth library.

Embedded Systems Example

In the following example, I use the FlexiPanel server C library to create a remote control panel for a data-logging embedded system. The library is intended to work with the lowest level embedded controllers possible. Therefore, I make no assumptions within the library about serial port buffering or multithreading support.

The data logger records the value of a proximity sensor. It has no UI of its own and relies on FlexiPanel to communicate with the outside world. The distance measured is displayed as a digital readout on a number control and as a historical log in a matrix control.

The data logger (Figure 5) is based on an 80186-based FlashCore embedded controller from Tern (http://www.tern.com/). A GP2D12 analog proximity sensor (Sharp Electronics) is connected to an A/D input and serves as the data source. A BlueWAVE Bluetooth serial module from Wireless Futures (http://www.wirelessfutures.co.uk/products/) is connected to a serial port to enable connection to FlexiPanel remote clients. The data logger's job is simple and few lines of code are required to implement it (Listing One). All calls to the FlexiPanel server library begin with the prefix FBVS. Like most FlexiPanel applications written for embedded controllers, it consists of fixed sections: initialization, user interface definition, main program loop, client message processing, and an ungraceful disconnect handler.

• Initialization. The Bluetooth stack and FlexiPanel server are initialized. This includes giving the UI a name. This name appears in a client's UI in the list of servers available for connection.
• User Interface Definition. Sandwiched between calls to FBVSStartControlList and FBVSPostControls, each control is defined by a call to an FBVSAdd... function. Each time FBVSPostControls is called, the previous UI definition is replaced; the UI can thus be changed at any time. FBVSSetOption lets an application request the way a control is depicted on a specific remote device. In Listing One, a point-plot chart is specified if the client is a PocketPC or a Windows computer.
• Main Program Loop. In the main program loop, the FlexiPanel library is polled for messages from client devices. Every second, the sensor is sampled and the sampled value is written to both the digital readout and historical log controls. In addition, every few seconds a ping test is made to check whether a client device is still in range. If a client was connected but contact was unexpectedly lost, control is passed to the ungraceful disconnect handler.
• Client Message Processing. When client-related events occur, the FlexiPanel server library posts notification messages to the application (Table 3). The data logger uses the FBVSGetNotifyCode function to collect these messages. Most notifications are informational and useful only for providing a local indication of the connection status. The four messages that the developer should always consider are:
• FBVSN_ClientConnected, a remote device connected to the server. In this example, the server clears the matrix control of old data.
• FBVSN_ClientData, where the remote device modified a control, users press a button, for instance. In this particular application, neither of the controls is modifiable by the client and so nothing needs to be done. Usually, however, the application would be expected to respond to any user interaction at this point.
• FBVSN_ClientDisconnected, where a remote device is disconnected from the server. In this example, the message is ignored.
• FBVSN_Abandon, an error occurred. This message has only ever been witnessed during application-device development due to a readily apparent programming error. In this example, as a precaution, control is passed to the ungraceful disconnect handler if this message is received.
• Ungraceful Disconnect Handler. The application must provide for the possibility that the connection is lost unexpectedly. This might occur because the remote device has gone out of range or its batteries are worn out. In this example, no action is necessary. In applications that control machinery, an emergency stop procedure should be implemented.

Windows .NET Example

The OrderMaster console application uses the FlexiPanel .NET library to create a remote control panel for a restaurant order-taking system. A Windows server computer with a printer is located near the kitchen and prints out the orders for the kitchen and the checks for the customers. It needs to be Bluetooth equipped and, therefore, will require a USB Bluetooth adapter (ideally a class 1 adapter with a 330-foot range).

The wait-staff carry remote devices for taking orders. The application is targeted specifically for PocketPC clients. The UI has been designed using a fat thumbs approach, where the controls are large enough that no stylus is required. In keeping with the FlexiPanel philosophy, however, any FlexiPanel client would be able to connect. A waiter who lost his PocketPC could always use his cellphone to take orders. Indeed, it would be possible for customers to place orders and print out their checks themselves if they had an appropriate device.

Figure 6 is the OrderMaster main screen. At the top is a text control for displaying a summary of the current order. Below it are section controls that drill down to specific sections of the menu. At the bottom is a list control to select the table being served and buttons to order the check, clear the order, and confirm the order.

Figure 7 is an Appetizer subscreen. At the top is the section control that returns to the main screen. Below it are number controls for setting order quantities for individual items on the menu. The code required to implement OrderMaster is available electronically; see "Resource Center," page 5. Since much of the code is repetitive and would in practice be replaced with loops working from a menu database, code for subscreens other than the appetizers has not been implemented.

The FlexiPanel .NET namespace is called RCapiM. It consists of static function calls for management of the user interface service and classes for each of the 12 control types. Like the data logger, this application consists of the same five sections: initialization, UI definition, main program loop, client message processing, and ungraceful disconnect handler.

• Initialization. The _tmain() entry point in midway through OrderMaster (available electronically) begins by initializing the FlexiPanel library, setting up the Bluetooth port, giving the UI a name, and setting up a timer to ping the remote device regularly. Since a PocketPC client is anticipated, several options are requested specifically for it. It should hide its usual navigation buttons and it should regularly ping the server (the computer, that is, not the waiter!).
• User Interface Definition. The UI is defined by creating controls and attaching them to an ArrayList called RemoteForm. The RemoteForm is then sent to the FlexiPanel::PostControls static function in order to display the control panel. If the application needs to respond when the user modifies a control, a delegate is added to the OnClientModify event for that control. Since the delegate will be called from a different thread to the main thread, the delegate contains static pointers to controls it needs to access. A delegate is also added to the static OnClientNotify event so that the order can be cleared if a Client Disconnected notification is received (see Table 3).
• Main Program Loop. In the main program loop, nothing happens except for waiting for the instruction to quit. All further activities are in response to events.
• Client Message Processing. The following delegates respond to events and are managed within the EvtHandler struct: OnClientNotify—if the client notifies it is disconnecting, the order is cleared; OnButCheck—if the Check button is pressed, the check is printed; OnButClear—if the Clear button is pressed, all order quantities are set to zero; OnButConfirm—if the Confirm button is pressed, the order is printed for the kitchen, then all order quantities are set to zero; OnCtlModify—if the Table list box is modified or one of the menu section controls is opened or closed, the status text box is updated; OnPingTimer—the client is pinged. If contact is lost, an Ungraceful Disconnect Handler procedure is followed.
• Ungraceful Disconnect Handler. The application clears the order and waits for the client to reconnect.

Future Developments

Born almost by accident while trying to optimize engine performance, the FlexiPanel Protocol has evolved into a patented user-interface service for a range of devices from tiny microcontrollers to .NET applications. Future development plans include:

• Embedding the protocol directly inside Bluetooth chipsets. This will lower the cost sufficiently so that even basic electrical appliances such as light switches can create remote user interfaces.
• Provision of a local Bluetooth to an HTTP bridge so that a web browser anywhere in the world might connect to a FlexiPanel server.
• Provision of a local Bluetooth to telecoms bridge so that a FlexiPanel server can be accessed by anyone dialing-in using a touch-tone phone.

DDJ

Listing One

// Bluetooth module is connected to serial port SER1
extern COM ser1_com;
#define BTH_BAUD 12 // 115,200 baud for SER1
#define BTH_INBUFF 1024 // SER1 Input buffer size
#define BTH_OUTBUFF 1024 // SER1 Output buffer size
unsigned char ser1_in_buf[BTH_INBUFF]; // SER1 Input buffer
unsigned char ser1_out_buf[BTH_OUTBUFF]; // SER1 Output buffer

// UI constants
#define CID_ATOD 1
#define CID_CHART_TY 2
#define NUM_CTRL 2
#define NUM_OPTION 2

// function prototypes
void ProcessFlexiPanelMessages(void);
void HaltProcesses(void);

void main(void)
{
   int16 loopcount;
   // initialize serial port (calls serial I/O library)
   s1_init( BTH_BAUD, ser1_in_buf, BTH_INBUFF,
      ser1_out_buf, BTH_OUTBUFF, &ser1_com );
   // initialize FlexiPanel library and give the UI a name
   FBVSInit( NULL, NULL, 1, &ser1_com, NUM_OPTION );
   FBVSSetDevNameAndCharSet( "Tern Demo" );
   
   // initiate control panel description
   FBVSStartControlList( NUM_CTRL );
   // numeric display of distance
   FBVSAddNumber( CID_ATOD, CTL_NUM_FIXEDPOINT, "Range", 0, 0, 0,
                                                  2, -2, "%% m", NULL);
      // graphical display of distance log
      FBVSAddMatrix(CID_CHART_TY, CTL_MTX_DATA_TY | CTL_MTX_Y_FIXEDPOINT |
              CTL_MTX_Y_2BYTE, "Data Log", 30, 0, NULL, 1, NULL, "Range",
              "Time", "%% m", "%HH%:%mm%:%ss%", 2, -2, 0, 0, NULL);
      // suggest how chart might be displayed
      FBVSSetOption( PPC_DEV_ID, CID_CHART_TY, PPC_ATT_STYLE,
                                           PPC_CST_MATRIX_POINTS );
      FBVSSetOption( WIN_DEV_ID, CID_CHART_TY, WIN_ATT_STYLE, 
                                           WIN_CST_MATRIX_POINTS ); 
   // complete control panel description
   FBVSPostControls( );
   // start control panel service
   FBVSConnect();

   // main program loop
   loopcount = 0;
   while (1)
   {
      // ensure each loop takes around 10ms
      delay_ms(10);
      // discover whether client has sent any messages
      ProcessFlexiPanelMessages();
      // every 3 seconds, ping
      loopcount ++;
      if (loopcount == 300)
      {
         loopcount = 0;
         // ping
         if ( FBVSIsClientConnected() && FBVSIsPingSupported() && FBVSPing() )
         {
            // lost contact with remote device; continue cautiously
            HaltProcesses();
         }
      }
      // every second, log proximity sensor
      if (loopcount%100==0)
      {
         int16 range;
         DateTimeU dt;
         // read A/D (calls A/D library)
         range = fb_ad16( 0xc6 );
         // update numeric display
         FBVSSetNumberControlData( CID_ATOD, range );
         // update chart
         SetToCurrentTime( &dt );
         FBVSAddMatrixControlData( CID_CHART_TY, &range, &dt );
         // send updated time to client
         FBVSUpdateControlsOnClient( );
      }
   }
}

void ProcessFlexiPanelMessages(void)
{
   // check for message
   switch (FBVSGetNotifyCode())
      {
         // nothing has happened
         case FBVSN_NoNotify:
            break;
         // Client has connected; clear the contents of the matrix control
         case FBVSN_ClientConnected:
            FBVSSetMatrixControlData( CID_CHART_TY, 0, NULL, NULL );
            break;
         // Client has modified a control. in this app, no controls are
         // modifiable by the client, so nothing to do
         case FBVSN_ClientData:
            break;
         // Client has disconnected
         case FBVSN_ClientDisconnected:
            break;
         // following messages are informational only and will be ignored
         case FBVSN_GotProfileRequest:
         case FBVSN_GotPinged:
         case FBVSN_GotPingReply:
         case FBVSN_GotAck:
         case FBVSN_IncompatibleVersion:
            break;
         case FBVSN_Abandon:
         // Error; generally only gets here during development
            HaltProcesses();
            Reset();
            break;
     }
}
void HaltProcesses(void)
{
   // In this function, anything controlled by the embedded
   // controller is put in a fail-safe state.
   // nothing being controlled in this app, so nothing to do
}