Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Channels ▼
RSS

C/C++

Controlling Motion-Tracking Devices


Mar99: Controlling Motion-Tracking Devices

Mike is director of engineering for InterSense. He can be contacted at mikeh @isense.com.


From 3D fly-throughs on the morning weather report to special effects in games and motion pictures, elements of 3D technology are cropping up everywhere around us. But as we start to interact with 3D virtual environments through games and simulations, we need new devices to drive this interactive experience. While joysticks, trackballs, and computer mice are fine for 2D interfaces, motion-tracking devices -- which report an object's position and/or orientation in real time as it moves -- are necessary to navigate 3D simulated worlds.

In flight simulator systems, for instance, the pilot wears a fully immersive (not see-through) headset with a screen in front of each eye. A computer generates a pair of images in front of each eye, creating a stereoscopic 3D scene. The 3D scene interactively changes as the pilot moves his head and controls (flies) the simulated aircraft. As the pilot turns his head, the motion tracker reports a change in position and orientation to the computer. The 3D scene in front of his eyes then changes, just as it would in the cockpit of his aircraft. The realism of the 3D graphical scene is only part of a successful simulation.

Correctly matching the pilot's head motion to the 3D scene motion is an absolute requirement for a successful simulation. If the scene moves when the pilot's head is stationary, or if there is a long lag between the pilot's head motion and the image motion, the simulation becomes ineffective. These factors, along with scene jitter, can cause discomfort (and even nausea) for users, defeating the purpose of the simulated experience.

Tracking technologies used for these applications are based on mechanical, magnetic, optical, acoustic, and inertial sensors to measure 3D motion in real time. Motion trackers from InterSense (the company I work for; http://www .isense.com/) combine sensor technologies through sensor fusion to produce low-latency, jitter-free 3D motion tracking. Trackers can be configured to track only orientation. We refer to orientation trackers as "three degrees of freedom" (3DOF). A "six degrees of freedom" (6DOF) tracker reports both position and orientation. In most applications, multiple trackers are used to track several objects. For example, a simulation system can track a user's hand as well as his head so the user can interact with the simulated environment. Figure 1 shows a typical simulation system put together with off-the-shelf components. Note that both the tracker (mounted on the ceiling) and the user's head-mounted display (HMD) are connected to a standard PC.

High-end flight simulation systems can consist of a $10,000 HMD, a $100,000 image generator, a $10,000 6DOF tracker, a motion platform cockpit, force feedback controls, and sophisticated custom software. As the technology matures, the components of these high-end simulation systems are becoming cost effective. A number of companies, such as Sony, have HMDs that cost about $800 with similar performance to headsets that sold for $10,000 just a few years ago. InterSense produces a 3DOF tracker, also for about $800, that gives high update rates with jitter-free outputs. Finally, graphics cards for PCs, as inexpensive as $200, come with OpenGL or DirectX hardware rendering functions and enough onboard video memory to produce a cost-effective simulation system.

Tracker Performance Issues

The same issues concerning tracker performance exist in both high-end and low-cost systems. How well the simulation works depends on how well the user's motion is tracked and matched in the images generated. The key areas of tracker performance are accuracy, jitter, latency, and latency jitter.

Accuracy is how well the system reports the true position and orientation of the object being tracked. This is a function of the type of use and how well the system is set up. How accurate a tracker you need depends on a number of things, but the biggest factor is whether you plan on overlaying the virtual world on the real world. For example, you might have see-through headsets for augmented reality, or you might track a glove the user is wearing to interact with real objects. In these cases, a fraction of a degree or inch might make a huge difference in how well the simulation works. In the case of fully immersive systems, the user does not have an external reference to check the accuracy of the tracker other than the horizon, which should be level when the head is level. A tracker accurate to a degree in orientation or an inch in position will probably work well.

Jitter is how much the reported position and orientation bounce around from the true or average value. A tracker that jitters will cause the generated scene to jump back and forth a couple of pixels every frame and cause much distress to the user.

Latency is the time lag, measured in milliseconds, between the instant when the user's head position is sampled and the instant when the corresponding rendered scene begins to scan out onto the display. Many things contribute to the lag, including the time it takes the tracker to generate an output, the time it takes to transfer the output to the renderer, and the time to render the image. Latency usually ranges from 16 ms to 120 ms.

Latency jitter is how much the latency changes from frame to frame. In some systems it might jump between one frame of delay to two. This is not noticeable when the head is still, but when it is moving it will cause perceptible jumps in the images.

Tracker Serial Interface Basics

The interface to most trackers is through a standard RS-232 serial communications port. Trackers usually support rates from 9600 to 115,200 bps. While the details of interfacing to each company's tracker varies, the basic principles are the same.

A tracker is set up to output data records in specified formats for the number of sensors needed. Then, it outputs a data record for each configured sensor as fast as it can (continuous mode) or when a data record has been requested (polled mode). In either case, the tracker's internal loop is running continuously and you output the data record after the sensors have been sampled and a new data record has been computed. Depending on how the tracker is set up, it can produce records at rates up to 150 Hz (for an IS-300) or 500 Hz (for an IS-300 PRO or IS-600). It is important to make sure the record sizes, types, and rates are fast enough to keep up with the required record throughput.

You can specify exactly what data you want in the data record for each station and its format. For reporting orientation, you have the option of choosing the Euler angles of yaw, pitch, and roll in degrees (3 floats); quaternions (4 floats); or direction cosines (9 floats). How the orientation is used in your code will determine if you want Eulers, quaternions, or direction cosines. The position can be reported as centimeters or inches for x, y, and z positions. All of the numbers can be reported in ASCII floats in notation, binary as 32-bit IEEE floating-point numbers, or in a compacted 16-bit binary format.

Example 1 is a stream of data records from a tracker for a single 6DOF station setup to report orientation in Euler angles and position in inches. Table 1 shows the number of bytes in a data record for a number of different record formats. Table 2 shows the transmission time of records for different record formats and baud rates. Time to transmit your data can vary up to 30 times depending on how you have your serial port set up and the format you choose for data transmission.

Listing One opens and configures the serial port under Windows 95. The only special note is that the reading of the port should be nonblocking. This is chosen because during the real-time segment of the program you can not afford to wait for input characters. Instead, you parse as much data as is available when you service the serial port in the main loop. Using the simpler blocking writes for the serial port is allowed when continuous mode is used because there is no need to send data during the real time part of the program.

Listing Two is a straightforward tracker initialization routine. It is used to make sure the data you get from the tracker is what you expect upon the start of every session. This puts the tracker in a known state prior to data initialization.

Serial Port Cautions

How well you write your serial port interface code affects the latency and latency jitter in the system. One key concept, often overlooked, is use of the most recent record available. Because the system buffers the data as it comes in the serial port, you want to make sure that when you read a record from the tracker you check to see if there are others available, reading through all of them. This check should be performed in both polled or continuous mode, even if you flushed the buffers at the start. Problems occur if, at the start of your code, you introduce a two or more frame delay in your serial input buffers. This produces unnecessary latency in the system.

You should do what checking you can on the data record. Since no checksum is sent with the data record, you should check for the O at the start, the station number, and that the correct number of bytes were received for this record. Otherwise, a corrupted record might cause a huge jump in the generated scene.

Direct3D Main Loop

Listing Three is the main loop from a simple DirectX Retained Mode full-screen program. It is a full-screen application for a number of reasons. First, it reduces the interactions with windows to get performance closer to real time. Second, it utilizes double buffering efficiently by performing page flipping to prevent tearing, thus allowing synchronization of the display. Finally, because this is going to ultimately be used in a headset, the image should take up the entire screen.

The main loop runs at the screen refresh rate set at 60 Hz if the rendering engine is fast enough to render its image in the given time. The loop starts by reading the serial port, then updating the user's viewpoint to match the latest record received. Any animations in the scene are updated a step and then the scene is rendered to the back buffer. Finally, when the vertical sync is detected, the flip is made between the back and front buffers.

Getting a Handleon Latency and Latency Jitter

The latency and latency jitter can be calculated for the Direct3D example. Assume the tracker is running in continuous mode asynchronously with the rendering loop running at 60 fps, and the tracker is in 3DOF mode running at 500 Hz in binary output mode at 115,200 bps. The data transmission time is 1.4 ms and the tracker is running at a loop time of 2 ms. In this case, the smallest latency could be 2 ms for the tracker to sample its sensors and produce an output record, plus 1.4 ms to transmit the results, and 16.7 ms for the PC to read the serial port and render the image, for a total time of 20.1 ms. The worst case latency would be if the PC just missed an update from the tracker when it read the serial port. This means its record would be 2 tracker cycles old=2 ms+2 ms+1.4 ms=5.4 ms, giving a total latency of 16.7+5.4= 22.1 ms. For this example, the latency ranged from 20.1 to 22.1 ms and the latency jitter is 2 ms.

I'll now turn to a 6DOF station running at 120 Hz. Even though at 120 Hz the loop time is 8.3 ms, you sample the inertial sensors and generate an updated output starting about 2 ms into the loop. The rest of the loop time is spent using the ultrasonic system to correct for any errors in the system. So, if the tracker is running at 120 Hz, then the latency could be from as little as 3.4+16.7=20.1 ms to 8.3 (one tracker cycle)+3.4+16.7=28.4 ms. For this example, the latency ranged from 20.1 to 28.4 ms and the latency jitter is 8.3 ms. If your head is panning at 100 degrees per second, this latency jitter manifests itself as 0.83 degrees of yaw jitter, which is very noticeable.

To fix the latency jitter for the tracker running at 120 Hz, you can genlock the tracker to the vertical sync of the monitor (Figure 2). Now the render loop will get its data from the tracker cycle that started 8.3 ms before the last vertical sync, giving a latency of 16.7+8.3=25 ms, but a latency jitter of close to 0 ms.

From these examples, it is clear that the two ways to eliminate latency jitter are to either use a tracker that can run at a very fast update rate or genlock the tracker to the monitor or image generator. Once you have fixed the latency jitter you can use prediction to eliminate some, if not all, of the constant latency. Because InterSense's motion tracker is based on measuring the user's accelerations and angular rates, we can directly predict the position and orientation of the tracked object from 0 to 50 ms ahead. This is better than simple dead reckoning prediction based on the last couple of positions and orientations, but it still will add some overshoot following abrupt motions, especially at values of greater than 30 ms.

One other major factor in latency jitter is that it is important to fix the scene complexity such that the rendering can be done in a fixed amount of time no matter what direction the user is looking. If the rendering takes too long it will drop the frame rate. If frames are dropped while the head is moving, the image will look like it is jittering.

Conclusion

To make the most effective immersive simulations with a given set of hardware, it is important to make sure that the interface to the motion tracker is working properly. And again, the key points for producing a successful simulation system are: writing an efficient serial port driver, using the appropriate data transfer formats, and elimination of the latency and latency jitter in a system.

DDJ

Listing One

#include <stdio.h>#include <string.h>
#include <windows.h>
#include <process.h>
#include <winbase.h>
#include <time.h>


</p>
BOOL rs232InitCommunications(HANDLE *portHandle, UINT32 comPort,
                                                       UINT32 baudRate);
BOOL rs232DeinitCommunications( HANDLE *portHandle);


</p>
//********************** rs232InitCommunications *******************
BOOL rs232InitCommunications(HANDLE *portHandle, UINT32 comPort, 
                                                         UINT32 baudRate)
{
    COMMTIMEOUTS timeout;
    DCB dcb;
    const char openString[100]="9600,N,8,1";
    rs232DeinitCommunications(portHandle); // close port if already open
    FillMemory(&dcb, sizeof(dcb), 0);          //now open the serial port
    *portHandle=CreateFile((comPort == 1 ? "COM1" : (comPort == 2 ? "COM2" :
                            (comPort == 3 ? "COM3" : "COM4"))),
                            GENERIC_READ | GENERIC_WRITE,
                            0, 0, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, 0);
    if(*portHandle == INVALID_HANDLE_VALUE)
    {
        printf( "Failed to open communications port");
        return FALSE;
    }
// Set the timeouts for nonblocking reads and blocking writes
    if (!GetCommTimeouts(*portHandle,&timeout))
    {
        printf( "Could not get timeout info");
        return FALSE;
    }
    timeout.ReadIntervalTimeout = MAXDWORD;
    timeout.ReadTotalTimeoutMultiplier = 0;
    timeout.ReadTotalTimeoutConstant = 0;
    timeout.WriteTotalTimeoutMultiplier = 10;
    timeout.WriteTotalTimeoutConstant = 100;


</p>
    if(!SetCommTimeouts(*portHandle, &timeout))


</p>
    {
        printf( "Could not set timeout info");
        return FALSE;
    }
        // now set up comm port with baud rate and other port parameters
    FillMemory(&dcb, sizeof(dcb), 0);
    dcb.DCBlength = sizeof(dcb);


</p>
    if (!GetCommState(*portHandle, &dcb))
    {
        printf( "Failed to get communications state");
        return FALSE;
    }
    BuildCommDCB(openString, &dcb);
    dcb.DCBlength = sizeof(dcb);
    dcb.BaudRate = baudRate;    // use the passed baud rate
    dcb.fNull = FALSE;
    dcb.fBinary = TRUE;
    dcb.fAbortOnError = FALSE;
    dcb.fOutX = FALSE;
    dcb.fInX = FALSE;
    dcb.fOutxCtsFlow = FALSE;
    dcb.fOutxDsrFlow = FALSE;
    dcb.fRtsControl = RTS_CONTROL_DISABLE;


</p>
    if (!SetCommState(*portHandle, &dcb))
    {
        printf( "Failed to set communications state");
        return FALSE;
    }
    return TRUE;
}
//******************** rs232DeinitCommunications *********************
BOOL rs232DeinitCommunications(HANDLE *portHandle)
{
    if(*portHandle != NULL)
    {
        CloseHandle(*portHandle);
        *portHandle = NULL;
    }
    return TRUE;
}

Back to Article

Listing Two

itSendCommand(portHandle,"O1,2,4,1\r\n");// setup output record format                                         // station 1 for Eulers and Pos
itSendCommand(portHandle,"f");           // binary mode
itSendCommand(portHandle,"U");           // Set position to inches
itSendCommand(portHandle,"l1,1\r\n ");   // Turn on station 1
itSendCommand(portHandle,"l2,0\r\n ");   // Turn off station 2 
itSendCommand(portHandle,"l3,0\r\n ");   // Turn off station 3 
itSendCommand(portHandle,"l4,0\r\n ");   // Turn off station 4
itSendCommand(portHandle,"C");           // Put the system in continuous mode
                                         //     to start sending data

Back to Article

Listing Three

void  Render(void){
RECT    rect;
DDBLTFX bltFx;
    updateTracker();    // read in the most current tracker data record
                        // use most current record to update users viewpoint
updateOrientation(Orientation[0][0],Orientation[0][1], Orientation[0][2]);
updatePosition(Position[0][0],Position[0][1],Position[0][2]);
    rect.left   = 0;    // Clear the buffer we are about to render to
    rect.top    = 0;
    rect.right  = 640;
    rect.bottom = 480;
 
    ZeroMemory(&bltFx,sizeof(DDBLTFX));
    bltFx.dwSize = sizeof(DDBLTFX);
    bltFx.dwFillColor = D3DRGB(1.0,1.0,1.0); // White background
    backBuffer->Blt(NULL,NULL,NULL,DDBLT_COLORFILL | DDBLT_WAIT ,&bltFx);


</p>
    // Update the direct3D Retained Mode scene
    scene->Move(D3DVALUE(1.0));
    view->Clear();
    view->Render(scene);
    dev->Update();
    
// Update primary surface by doing a flip when vertical sync comes along
    while(frontBuffer->Flip( NULL,0 ) == DDERR_WASSTILLDRAWING);
}

Back to Article


Copyright © 1999, Dr. Dobb's Journal

Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.