Wireless Data and Minimum Airtime Software
Cutting down on data-transfer errors
Darrell Diem
Darrell is software development manager for Motorola EMBARC advanced messaging. Darrell can be contacted at 407-463-3151
Wireless networks can be implemented as either LANs, MANs (metropolitan area networks) or WANs (wide area networks). By providing coverage for a building, city, region, or even a nation, wireless-network technology makes it possible for you to stay connected while still on the move. However, software developers who wish to create wireless-enabled applications need to understand some of the technical problems and user limitations of wireless networks.
The biggest limitation of wireless networks is the range and signal strength of transmitters. The location of the receiver in relation to structural surroundings can also affect reception quality. Some radio signals do a better job of penetrating buildings and earth than others. Wireless networks that use paging technology, for example, provide for signal overlap, and the receivers are designed to catch even the faintest signals. But in many cases, an end user can do little to improve the likelihood of error-free data reception--other than change his or her location.
Even if the receiving unit is within the transmitter's range and the signal comes through clearly, problems such as electrical interference (either atmospheric or man-made) can corrupt a file during wireless transmission. Often, the entire file needs to be transmitted again. This does more than just frustrate the person waiting for information in the field--it becomes a costly way to manage wireless data.
Obviously, the less time the data spends flying across the air, the better the chance for error-free reception. All else being equal, the time it takes to move data across a network is directly related to the amount of data being moved. Of course, other variables apply to this equation, like data compression and throughput speed. But in general, small files take less time to move across networks, regardless if they're wired or wireless.
When coded into applications, intelligent data-management software techniques, give the user control over the amount of time data spends in the air. This not only reduces the chance for errors, but significantly reduces a users bill for wireless-network airtime. These approaches incorporate "minimum airtime software" (MATS) algorithms.
Airbase, the MATS implementation described in this article, was developed for Motorola's EMBARC (Electronic Mail Broadcast to a Roaming Computer) nationwide, wireless e-mail and data network. EMBARC, a one-way wireless approach to roaming data, is based on paging broadcast technology. You can use EMBARC to send e-mail, transfer files, and update databases simultaneously to one or more users who may be widely dispersed--and it costs the same no matter how many people you send the message to.
Mobile computer users need low-error wireless delivery of data that they can control, and they need the ability to quickly recover corrupted or lost data. These needs led to the development of an "intelligent packet" MATS solution that addresses the problems associated with lengthy over-the-air transmission time.
MATS: A Practical Database Example
Most applications carried by mobile PCs don't require the transfer of large amounts of data--usually the bulk of the data is already on the PC and only needs to be "freshened." Attempts to send a full file over-the-air when only a small part has changed results in higher costs and greater chance for error.
MATS first assumes that common data is dispersed. It then updates to that database via packets containing new information along with instructions on how to process it into this roaming copy of the database.
Sending data in small packets minimizes exposure to interference-induced errors. Since packets are also identified as a part of a larger "set," they can be recombined into their original size and format on the receiving platform (usually a PC-type device). Error checking and ID tracking on each packet provides the user with information on a missed or corrupt packet. The receiving person or system can request retransmission of a missing or corrupt packet.
However, multiple errors in a file can still occur if a large number of packets are sent and rebuilt by the receiving application. An error in an ASCII e-mail file is usually tolerable since the document is read by a human, who can usually interpolate the correct answer from the context.
Binary file transfers are a different story. An 1-byte error can crash a program or change a significant digit in a database. Since databases tend to be large (frequently running into multi-megabytes), certain methods must be used to minimize over-the-air time. In addition to reducing the chance for random externally induced errors that can trash a file transfer, these methods are also less expensive than retransmitting the complete file.
Figure 1 illustrates the complex interrelationship of a number of variables, all contributing to message errors or corruption. Length of message, mobility of receiver, and signal strength are some sources of externally induced errors.
The MATS approach we implemented to solve these large data-transfer problems involved the development of a DBF (xBase) file editing and packetization software tool called "Airbase." The tool, which creates a standardized packet with the changes made only to each DBF record in a file, provides a means of launching a batch process upon receipt of the data, and identifying and recovering any lost or corrupt packets. (Airbase is available at no charge from the author.)
Creating Packets
In the editing, or update, process, the server-based user selects a DBF file to update, and Airbase creates a "shadow" or companion database file. The names of the databases are the same except that the extension on the shadow copy is DB$. Figure 2 shows how the structure of the shadow file differs from the original. Notice the two additional fields in the DB$ structure. All other fields match the DBF structure. The ACTION field in SAMPLE.DB$ is a 1-byte character field containing the intended edit activity: add, copy, and delete. The edit process copies all data from an edited record into the DB$ file.
The specific edit process of adding, updating, copying, and deleting causes the appropriate character to be placed in the ACTION field of the DB$ database. The sort, or key-field, data of the original SAMPLE.DBF record is also placed in the K_FIELD field of DB$. Figure 3 shows sample record and its DB$ shadow. This data represents an update to the LAST_NAME, ADDRESS and to the CITY/ST/ZIP fields. The sort field is LAST_NAME and the sort data is "Smith." The record in the SAMPLE.DBF database isn't changed at this point. Changes are accumulated in the shadow file until the session is complete or until the Transmit option is selected from the edit menu. This means that all databases are the same, including the master database, until Transmit is invoked, at which time the following happens:
- Each record in DB$ is compared to the original in the DBF file.
- Each field with a change is extracted into a buffer.
- A packet is created for each changed record.
- The packet is submitted for transmission.
- The packet is itself placed in a transmission packet and transmitted.
- The file SAMPLE.DBF record is updated.
- The SAMPLE.DB$ record is marked "sent" (deleted with *).
Receiver Processing
The packet is received by a wireless receiver (the NewsStream receiver, in the case of EMBARC) and stored in the unit's internal buffer. When connected to a PC equipped with receiving software, the packets are downloaded to mailboxes and stored on disk. The system-level transmission packet header contains information including its own CRC error-checking data, source of message, time sent, number of pieces to come, application tag, unique ID, and so forth.
The download software uses the application tag to determine if the packet is an Airbase data packet. If the download software is installed with the auto-process flag set to "on" and, if an Airbase packet is received, upon processing the last packet, the software, will stuff the keyboard buffer with a numerically named batch file. If no foreground process is running, this will launch the batch file for postprocessing the packet according to the batch-file contents. This could include a dial-back request for lost or corrupt packets to be re-broadcast. Listing One (page 102) shows how postprocessing is implemented. The first byte of the packet is the batch-file number. For example, a "5" in the first byte will launch a batch file named 5.bat, and so on. If the first byte is a 0, no keyboard stuffing occurs. Listing Two (page 103) provides details on download processing.
User Control
Recipients of the data can use Airbase to review the mailbox data and process it into a target database, delete the data, launch the indicated batch file, or do nothing.
The received data-packet header is used to find the database file using the pathname in the packet. This lets the sender and user to update or maintain multiple copies on the receiving platform. The recipient controls access to the data by installing the wireless-modem download software with the auto-process flag set on or off. Manual execution of updates can than be executed via menus.
Airbase is a three-way application: It's an editor and wireless transmit package, it provides manual review and update options for received data, and it can be called with command-line processing strings in a batch-file mode for auto-updating. See Listing One for details on how to process the packet. (Airbase, a free library, and sample source for this process are available from the author.)
Conclusion
The benefits from a small packet of "difference only" data such as that provided by MATS are:
- Reduced opportunity for random errors due to interference,
- Receiver "overview" of data before application of updates to a data file
- Identification of lost/corrupt packets
- Lower cost of transmission.
Other approaches to reducing the airtime for over-the-air file updates are required when the data structures are too complex to address with a special-purpose application such as Airbase. Spreadsheets and word-processing files are two such examples. To minimize file size and airtime exposure, the files could be compressed but, in most cases, they will still be large. Also available are utilities that create a "difference" file by comparing an original with a changed file. One such "differencing" package is .RTPatch by Pocket Soft (Houston, Texas).
Coverage, error management, and the cost of wireless data distribution will improve over time. The users of the technology need to know the strengths and limits of any given implementation in making choices. Software selection and data-management strategies are opportunities for users to improve delivery quality and reduce costs. Management of data before, during, and after wireless delivery are equally important. A little thought can provide a higher-quality, cost-effective solution for maintaining "roaming data."
Figure 1: Variables that can contribute to message errors or corruption.
Figure 2: The structure of the original (SAMPLE.DBF) vs. a shadow file (SAMPLE.DB$).
Figure 3: A sample record (SAMPLE.DBF) and its DB$ shadow (SAMPLE.DB$).
Table 1: Airbase data-packet structure
NAME Bytes TYPE CONTENTS
Batch Tag 1 Binary Batch-file tag number 0x01 = no
tag used
Trans Type 1 ASCII Transaction Type:
A ADD
U Update
C Copy
D Delete
Time Stamp 17 ASCII Nonnull terminated time stamp.
String format:
"MM/DD/YY<space>HH:MM:SS".
Num of Fields 1 Binary Number of fields in the packet body
BCC 1 Binary Block -check character. Calculated
over entire packet except for the
BCC component.
Dest DB Name 1 Binary Length of destination DB,
Length 64-character maximum.
Dest DB Name 0 to 64 ASCII Pathname of destination DB, nonnull
term.
Field Name 1 Binary Length of field name.
Length
Field Name 1 to 25 ASCII Field name of changed data, nonnull
term.
Key Index 1 Binary Key indicator level of field:
0x00 not a key field
0x01 primary key field
0x02 secondary key field
0x03 tertiary key
0xnn 0xnn<I>th</I> key
Data Length 2 Binary Length of data.
Data 0 to 64K ASCII/
Binary Actual data.
_WIRELESS DATA AND MINIMUM AIRTIME SOFTWARE_ by Darrell Diem [LISTING ONE] /************************************************************************** ** FUNCTION: MasterDBUpdate ** DESCRIPTION: Dummy master database update routine invoked by the ** TSR record handler for live updates ** PARAMETERS: None ** RETURN VALUES: ** ERRCTKNONE - Master D/B update performed successfully ** ERRxxx - Master D/B update not performed. The Message Handler ** will write the record to the Holding File ** ERROR HANDLING: ** NOTES: ** PSEUDOCODE: ** BEGIN MasterDBUpdate ** END MasterDBUpdate ***************************************************************************/ int DBUpdate(int iApiIdTag, int Flag_argc) { int iRecNum,icounter; char cTransType; int iErCode = 0; int iRetn = 0; int iNumFields; MBOX *pstMbx; // pointer to mailbox structure created by API MBOX_MSG_HDR *pstHdr; // buffer for mbox hdr data static char szOldKeyVal[FLDVALSIZ + 1]; static char szNewKeyVal[FLDVALSIZ + 1]; static char szTimeStamp[CTKTIMESTAMPLEN]; static char szDataFile[_MAX_PATH]; static char szIndxFile[_MAX_PATH]; static char szDrive[_MAX_DRIVE]; static char szDir[_MAX_DIR]; static char szFname[_MAX_FNAME]; static char szExt[_MAX_EXT]; PDBINDFILEX pstDbFileX; /* perform CtkBeginTrans to get the Transaction Type, the Time Stamp, ** the Number of Fields in the Client/Server Packet, and the Destination ** Database Name */ CtkQueueSize(iApiIdTag,&iRecNum); icounter = iRecNum; //number of records in mbox if(Flag_argc == 0) //flag to do only newest data iRecNum = 1; //newest 1st so updates reflect only latest data //command line overrides the num recs selection if(Flag_argc > 0 && Flag_argc != 'X') { iRecNum =Flag_argc; Flag_argc = 0; //do only indicated rec Number //flag_argc to break while loop 1st pass } /* loop until all processed or break encountered */ while (iRecNum > 0 && iRecNum <= icounter ) { /* open mailbox to check app_flag processing status 3 == done */ if ((iErCode = API_open_mbox(iApiIdTag, MBX_RD_WRT, &pstMbx)) ! = EPC_NO_ERR) { return iErCode; } /* retrieve mailbox message header for this message and get message length */ if ((pstHdr = malloc((size_t) sizeof(MBOX_MSG_HDR))) == NULL) { API_close_mbox(&pstMbx); return ERRCTKMEMALLOCHDR; } if ((iErCode = API_get_msg_hdr_from_mbox(pstMbx, iRecNum, pstHdr)) != EPC_NO_ERR) { free(pstHdr); API_close_mbox(&pstMbx); return iErCode; } iRetn = pstHdr->app_flag; //if zero set app_flag to API_set_app_flag(pstMbx,iRecNum,3); //num 3 for 'processed' data free(pstHdr); API_close_mbox(&pstMbx); //this process is for command-line updates if(iRetn >= VIEWED && Flag_argc > 0) { iRecNum--; if(iRecNum <= icounter) break; continue; } CtkReadQueue(iApiIdTag, iRecNum); //set up data packet buffer if ((iRetn = CtkBeginTrans(&cTransType, szTimeStamp, &iNumFields, szDataFile)) == ERRCTKNONE) { /* construct index file name from the name of the data file by ** changing the extension from (nominally) ".DBF" to ".NDX" */ _splitpath(szDataFile, szDrive, szDir, szFname, szExt); _makepath(szIndxFile, szDrive, szDir, szFname, ".NDX"); /* attempt to open the data and index files for writing */ if ((pstDbFileX = DbOpenX(szDataFile, szIndxFile, TRUE)) == NULL) { iRetn = ERRCTKDBUPDATEFAILED; } else { /* attempt to retrieve the record from the database */ iRetn = LocateDbRec(pstDbFileX, iNumFields, szOldKeyVal, szNewKeyVal); switch(iRetn) { /* record was found in D/B */ case ERRDB3NONE: switch (cTransType) { case CTKTRANSADD: iRetn = ERRCTKDUPLICATEREC; break; case CTKTRANSUPDATE: iRetn = TransUpd(pstDbFileX, iNumFields, szOldKeyVal, szNewKeyVal); break; case CTKTRANSDELETE: iRetn = TransDel(pstDbFileX); break; default: iRetn = ERRCTKINTERROR; break; } break; /* record was not found in D/B */ case ERRDB3READFAILED: case ERRDB3SEARCHFAILED: switch (cTransType) { case CTKTRANSADD: iRetn = TransAdd(pstDbFileX, iNumFields, szNewKeyVal); break; case CTKTRANSUPDATE: case CTKTRANSDELETE: iRetn = ERRCTKNOSUCHREC; break; default: iRetn = ERRCTKINTERROR; break; } default: break; } /* close the data and index files */ DbCloseX(pstDbFileX); } /* end transaction, and if it failed then record error */ if ((iErCode = CtkEndTrans()) != ERRCTKNONE && iRetn == ERRCTKNONE) { iRetn = ERRCTKDBUPDATEFAILED; } iRecNum--; if(iRecNum <= 0 || Flag_argc == 0) break; } //end while loop API_enable_core(API_NO_FUNCT); } fclose(pstTraceFile); return iRetn; }[LISTING TWO]
<a name="038c_000d">
// Batch file stuff goes here
//
#ifdef BATCH_STUFF
if(iBatch_Tag == 1) {
if ( MbxHeader.api_tag >= 33 && MbxHeader.api_tag <= 39 )
{
if ( do_batch_thing != 0 )
{
char batch_key_stuffin[30];
// the first byte of the message is called do_batch_thing
_itoa(do_batch_thing, batch_key_stuffin, 10);
strcat(batch_key_stuffin,".bat\r\n");
kbflush();
if(MbxHeader.k_block == MbxHeader.n_blocks) //use only last request in list
kbstuff(1, batch_key_stuffin);
}
}
Copyright © 1993, Dr. Dobb's Journal
