Monitoring Data in Real Time
Worldwide access via an Internet connection
Tony Garcia, Jeff Woodard, and Craig Liddell
Tony is system administrator and Craig is associate professor in the plant pathology laboratory at New Mexico State University. Jeff was formerly a computer operations supervisor at the university. They can be contacted at [email protected].
One of the most poorly monitored conditions in biological-research laboratories is the temperature fluctuation of incubators, freezers, and cold rooms used for routine storage and incubation of samples and cultures. Most biologists do not need the custom, highly automated, precision, temperature-monitoring systems commonly used in industry and physics/engineering laboratories. These systems are generally too expensive and overspecified for our needs. Furthermore, they often require more complex maintenance than we are able to provide.
We know from bitter experience that drifting temperatures can cause many problems in a biological laboratory, because the growth of most organisms is highly temperature dependent. We have experienced unexpected results in experiments using standard incubators or by depending upon the ubiquitous "room temperature" which, in our facility, can vary from 18 to 25 degrees Celsius. Our laboratory often cools to the point of causing precipitation in reagents and condensation inside microscope optical assemblies. And the fact that room temperature can vary depending on season or climate occasionally leads to conflicting results from different laboratories.
However, inexpensive PCs, desktop workstations, and general-purpose data-logging hardware make it possible for researchers to develop a real-time system that monitors temperature. We built a system that not only monitors temperature, but also alerts laboratory personnel of any unusual fluctuations. Our specific goal was to build a system that could display temperatures large enough to see from any location in the laboratory and that could provide unequivocal alert and alarm conditions. We used archival temperature records to help us solve intermittent and periodic problems with temperature control, and we appended temperature data to World Wide Web documents to make remote access to our lab's data possible. Our "Growth Chamber Alarm (GCA)" system is purposely simple, and allows even the most computer-illiterate personnel to set temperature limits and label incubators.
The design criteria, of course, became complex as we developed the GCA system. Since some chambers cycle between two set points over 24 hours and others contain organisms growing at temperatures near their low or high cardinal limits, we needed to be able to set upper and lower temperature bounds independently for each monitored chamber. We also had to distinguish between temporary temperature fluctuations (like those due to opening the door of an incubator) and real deviations that required attention. Finally, we had to monitor certain system-level functions that indicate actual or impending failure of the system. The major system-level considerations were communication failures (at any point in the system) and power supply.
The Hardware
The design criteria for the GCA system were determined by implementation decisions that more or less evolved as the system was developed. We decided early to go with off-the-shelf hardware and software for as much of the system as possible to save time and money. However, many of our primary design considerations could not be met by commercial systems without highly expensive, third-party custom development. We chose to use the CSI CR10 Measurement and Control Module with a CSI AM416 multiplexer that provides 64 single-ended channels for Type T (copper-constantan) thermocouples. (Both items are available from Campbell Scientific in Logan, UT.) We currently monitor the temperatures of 14 chambers in the laboratory including benchtop incubators, plant growth chambers, freezers, refrigerators, and several rooms in the facility using 18 thermocouple probes. We can expand the number of probes to 64 using the AM416 as needed; see Figure 1.
The CSI hardware and software, which is designed to monitor analog and pulse data, is inexpensive and easy to set up. The CR10 datalogger has 64 KB of on-board RAM and stores up to three or four months' data, depending upon the sampling rate and averaging periods chosen. We use a rechargeable 12-volt, lead-acid battery as a backup power supply (independent of the building's power system). The lead-acid battery is on a 12-hour charge cycle.
The CR10 used by GCA is programmed to scan the probes every 30 seconds and store hourly averages in final memory, which is volatile, circular, and uses most of the on-board RAM when the datalogger memory is full. Campbell Scientific supplies software that uses RS-232 communications with the CR10 and allows monitoring and downloading of both input storage values (30-second values) and final storage values (hourly averages). The CSI software also allows synchronization of the CR10 clock with the PC clock, and other facilities to program and debug the CR10 itself. The only other dedicated hardware in the system is an ISA-bus 486/33 PC running Novell DOS 7.0 used to communicate with the CR10, display the temperatures, alerts and alarms, and provide connection to the TCP/IP Internet link.
Getting the Temperature
The main software component, called GCA95, resides on the dedicated PC (see Figure 2). GCA95 was written to give the highest priority to maintaining current temperatures on the PC screen and displaying any alarms or alerts requiring attention. The console display uses colors and flashing signals, allowing experimenters to identify out-of-range alerts and alarms from nearly any workbench location in the lab.
GCA95 communicates directly with the CR10 using low-level serial communications (see Listing One) to download data stored in the input storage location of the CR10. The call_logger procedure in Listing One acts in the CR10 Remote Keyboard State by sending unsigned ASCII characters over an RS-232 line and uses the same command set as the CR10 keyboard. This keyboard is a small keypad, similar to building-security keypads, that uses a highly cryptic command set to communicate with the CR10. The call_logger procedure downloads and provides the data to update current temperatures on the PC screen every 30 seconds.
Once an hour, GCA95 shells to DOS and runs the CSI Telcom program to download data in the volatile final storage of the CR10 that contains the hourly average temperatures. This temperature summary is saved in a file on the local hard drive, printed on a nearby printer, and transferred across the network to our UNIX Web server once an hour. By running a Perl script on our Web server, we are able to maintain a World Wide Web page containing the last 24 hours of temperature data for each chamber and archived temperature data from the last four months to the present. Users at the PC console can access a user screen that allows the viewing of a graph of the last 24-hour temperatures for each chamber and an alarm history of all chambers.
Custom Software
The GCA95 program was written in C++ and uses two different classes: one is the interface, and the other, the interpretive engine (see Figure 2). We tried to keep the classes as independent as possible to simplify upgrades. The engine class performs 90 percent of the interpretation of data. The engine allows the interface class only minimal access to reduce the occurrence of logical bugs and to allow major changes to take place in the interface without interfering in data interpretation.
The engine class contains the communication functions and the alarm monitoring functions. GCA95 communicates directly with the CR10 using low-level serial communications to maintain current temperatures on the PC console. The call_logger procedure downloads the real-time temperature of each probe into a float array named corks[]. The values in this array are subsequently checked for changing conditions and for out-of-range temperatures or voltages.
The interface class displays each chamber with a label, temperature, and arrows indicating rising or falling temperatures. Each monitored temperature has a unique label and an upper and lower bound that the interface uses to color code each chamber on output to the screen. An ASCII file, init.dat, contains the label for each probe and the upper and lower bounds specified for each temperature. The chambers are displayed on the screen in the user-specified order contained in the file. The ability to manipulate display order allows grouping of similar chambers such as freezers or room-temperature probes. GCA95 initializes from init.dat at startup. This enables users to easily change labels, temperature bounds, and display order by editing init.dat with a text editor. If, for some reason, this file does not exist, GCA95 initializes itself from a default set and creates it.
There are four alarms displayed by GCA95: low battery voltage, building-power failure, communication failure, and out-of-range temperature alarm. All the alarms flash alternately between a bright yellow circle and bright yellow text that identifies the alarm. This color scheme provides a high degree of contrast against the black "starry night" background, allowing researchers to easily identify alarms. Whenever a temperature alarm occurs, the chamber label is written to a file that maintains a list of alarms in case of a power failure. This file is read upon startup of GCA95 to restore to normal any alarms that had been on prior to the power failure.
The voltage of the backup lead-acid battery must necessarily be monitored for low conditions to prevent a power loss to the CR10 that would potentially cause the loss of all volatile memory, including program instructions. Voltages below 11.76 also could damage the battery. The low-battery-voltage alarm occurs immediately upon falling below 11.76 volts.
In addition to monitoring the voltage of the backup battery, building-power failures are frequent enough to warrant consideration in GCA95 code. This is because a long power outage will directly affect the ability of all chambers to maintain temperature and will shut down the whole GCA95 system, except for the CR10 on battery backup. The building-power-failure alarm is implemented in the code with the overloaded function power_ failure_check(), which creates a zero-byte file called OK whenever the main program exits normally. Upon startup of GCA95, power_failure_check() erases the OK file (if it exists) and continues execution. If OK is not found, the function trips the building-power-failure alarm. Of course, an abnormal termination of the program due to crashing will also cause the building-power-failure alarm to trip.
Communication with the CR10 is obviously important to the system, and an alarm is triggered whenever there is a failure to communicate with the CR10. The failure is detected by a lack of response from the CR10 when attempting to establish serial communications. The call_logger() function sends an initiation value, 0Dh, to the datalogger, then checks for the acknowledgment value, E000. call_logger() attempts to initiate communications ten times before setting the comm_error flag to True in the error_flags[] alarm-status array.
After each 30-second status update, monitored temperatures are checked for out-of-range conditions. If the temperature is in range, the display is green; if it is above range, the display is flashing red; and if it is below range, the display is flashing blue. These alerts respond in real time to the actual temperature and are transient. When a chamber breaks a bound, its number is added to behave_array[]. If a chamber number exists in behave_array[] for longer than 50 minutes, a temperature alarm is tripped. This 50-minute out-of-range allowance is to avoid false alarms caused by opening of incubator doors.
An important backup to the alarm system is the 24-hour graphing feature. The user may view a graph of the previous 24-hour temperature data for each chamber; the graph also indicates the high and low temperature ranges for the chamber being viewed. The data is read from a file that is updated once an hour from the CR10 by the CSI Telcom program. The file consists of a date and time stamp and the average hourly temperature for each chamber. Since every hour is recorded, this file can become rather large if not archived periodically. We accounted for this problem when getting data to graph by reading each hourly temperature into a circular queue containing only 24 elements with a pointer to indicate the next available space. The pointer is incremented after each current value is read into the array, rather than shuffling array memory. Once the end of file is encountered, the values in the array are simply graphed from the oldest to newest data.
Run Time
GCA95 starts its hourly routines at three minutes past the hour to allow for any clock drift between the CR10 and the PC. The first hourly task is to download the previous hour's averages and append these to the data file. The program shells to DOS and executes Telcom as described earlier. At four minutes past the hour, the file is read by GCA95, and subsequently prints a hard copy of the last hour's temperatures. Control is immediately given back to the main loop after each of these tasks to prevent any interruptions in the refresh of GCA95's current status.
At five minutes past, during the next idle period, GCA95 reads the data file and writes the temperature data from the last 24 hours to a file called current.tmp. Then a DOS shell is executed that runs a batch file to copy current.tmp to a logical PCNFS drive mounted to a writeable, exported directory on our Web server. The final housekeeping task performed by GCA95 is to copy the entire data file over to the mounted UNIX drive.
At ten minutes past every hour, the crontab on our UNIX machine executes a Perl script (Listing Two) that processes current.tmp and the data file. First, current.tmp is converted from DOS to UNIX text format by stripping line feeds. The script then sends the data file through a filter written in C++ (listing available electronically), which outputs the text in an HTML table format. The script copies the converted document into a directory (readable by the HTTP daemon). The data file is converted to UNIX format and also placed into the Web directories on our Web server at http://taipan.nmsu.edu/pathlab/c.html.
The data file is updated every hour along with the previous 24-hour data, so any problems discovered when viewing the last 24-hour page can be checked with pages from the last several days or weeks by viewing the HTML archive.
Conclusion
The growth-chamber alarm-monitoring system we have designed has proved useful in allowing researchers either in the lab or at any networked computer in the world to monitor the progress of their experiments. The GCA system has increased the speed with which researchers respond to malfunctioning growth chambers and has vastly improved the ability of on-site researchers to monitor temperatures throughout the laboratory facility. In the future, we may control the actual thermostats with GCA95, making the system highly integrated. Also, we will soon add queries to the HTML pages to allow lookup of specific chambers as well as specific days.
Listing One
Listing Two
/****************************************************************************
** Procedure: Call_Logger
** Purpose: Calls the CR10 datalogger and gets values of current temperatures.
*****************************************************************************/
int targets::Call_Logger (){
unsigned dataval, status;
int loop;
_bios_serialcom (_COM_INIT, 1, _COM_CHR8 | _COM_STOP1 | _COM_NOPARITY | _COM_9600);
loop = 0;
status = 0xe000;
dataval = 0;
/* try to call the logger 10 times or until it responds */
while ((loop < 10) && (status == 0xe000)) {
status = _bios_serialcom (_COM_SEND, 1, 0x000d);
status = _bios_serialcom (_COM_SEND, 1, 0x000d);
status = _bios_serialcom (_COM_SEND, 1, 0x000d);
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
loop++;
} /* end while */
/* if the initial connection to the logger has failed, then exit */
if (status != 0x000a) {
error_flags[comm_error] = 1;
return(-1);
} /* end if */
status = _bios_serialcom (_COM_SEND, 1, (unsigned) '7');
status = _bios_serialcom (_COM_SEND, 1, (unsigned) 'H');
status = _bios_serialcom (_COM_SEND, 1, 0x000d);
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
status = _bios_serialcom (_COM_SEND, 1, (unsigned) '*');
status = _bios_serialcom (_COM_RECEIVE, 1, dataval); /* M */
status = _bios_serialcom (_COM_RECEIVE, 1, dataval); /* O */
status = _bios_serialcom (_COM_RECEIVE, 1, dataval); /* D */
status = _bios_serialcom (_COM_RECEIVE, 1, dataval); /* E */
status = _bios_serialcom (_COM_SEND, 1, (unsigned) '6');
/* gets unneeded characters sent by the CR10, and alligns read sequence */
for (loop = 0; loop < 24; loop++) {
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
} /* end for */
/* actually reads the required values from the CR10 in *6 mode */
for (loop = 0; loop < 19; loop++){
Get_Value (loop);
}
/* logs off from the keypad mode of the CR10 */
status = _bios_serialcom (_COM_SEND, 1, (unsigned) '*');
status = _bios_serialcom (_COM_SEND, 1, (unsigned) '0');
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
status = _bios_serialcom (_COM_SEND, 1, (unsigned) 'E');
error_flags[comm_error] = 0;
return(1);
} /* end Call_Logger */
/*****************************************************************************
** Procedure: Get_Value
** Purpose: This procedure reads and converts the values obtained from
** logger as they come across the com port.
** Return Value:
****************************************************************************/
void targets::Get_Value (int count){
char value[10];
int loop;
unsigned dataval, status;
dataval = 0;
/* sends the 'A' keyboard command to CR10 */
status = _bios_serialcom (_COM_SEND, 1, (unsigned) 'A');
/* reads the cr lf sequence sent by logger in response */
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
/* reads the location label from CR10 and throws it away */
for (loop = 0; loop < 3; loop++){
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
}
/* reads the actual number value from the logger */
for (loop = 0; loop < 7; loop++) {
status = _bios_serialcom (_COM_RECEIVE, 1, dataval);
value[loop] = (char)status;
} /* end for */
value[7] = '\0';
/* assigns the old value, and sets the new value */
last_cork[count] = corks[count];
/* checks to make sure an acceptable value is being assigned */
if ((atof (value) > -999) && (atof (value) < 999)){
corks[count] = atof (value);
}
} /* end Get_Value */
#!/usr/local/bin/perl
$old_name = "/taipan3/tonyself/current.tmp";
$new_temp = "/taipan3/tonyself/c.tmp";
$new_name = "/taipan5/pub-html/pathlab/c.html";
$html_header = "/taipan5/pub-html/pathlab/html.tags";
$gc = "/taipan3/tonyself/gc.dat";
$gctemp = "/taipan3/tonyself/gc.tmp";
$html_head2 = "/taipan5/pub-html/pathlab/html2.tags";
$gchtml = "/taipan5/pub-html/pathlab/gc.html";
if(-e "$old_name"){
system "dos2unix", $old_name, $new_temp;
system "/bin/cat $html_header $new_temp > $new_name";
system "/bin/chmod 644 $new_name";
}
if(-e "$gc"){
system "dos2unix", $gc, $gctemp;
system "/bin/cat $html_head2 $gctemp > $gchtml";
system "/bin/chmod 644 $gchtml";
}
