Examining QNX RTOS 6.1

QNX's major downside was that it was available for the Intel x86 platform only, while other processors were more prevalent, especially in the embedded systems market. That's why QNX Software Systems decided to completely redesign the kernel. The new kernel, named "Neutrino," was designed to support multiple platforms — MIPS, PowerPC, StrongARM, and the like — and released in 1996. Today, all releases of the QNX RTOS, including Version 6.1, use this kernel.

Architecture of a Modern RTOS

The QNX RTOS 6.1 is a message-based operating system with a true microkernel architecture. Message passing is the primary means of interprocess communication. Most of the API calls use this message-passing mechanism. For example, when an application wants to open a file, the system call is translated into a message that is sent to the filesystem manager. The file manager (after accessing the storage device via its device drivers) replies with a file handle. Since this message-passing mechanism is network transparent, the system in its entirety can be seamlessly distributed over several nodes, without requiring any modifications in the application code.

The microkernel architecture fosters robustness and stability. Every system manager, device driver, and user process runs in its own private virtual memory space, protected from all other processes. The QNX RTOS is also what's called "High Availability (HA) capable," a term that describes an RTOS's ability to stay up and running without interruption for extended periods of time. The QNX RTOS has inherent HA features, such as virtual memory protection and the capability to dynamically load/unload system components. More recently, QNX has added HA-specific modules that support hot swapping and recovery whenever system services, processes, and server connections fail.

Testing the System

We submitted the QNX RTOS 6.1 to our real-time performance and robustness test suite, which addresses thread and process handling, advanced interrupt handling, synchronization mechanisms, filesystem, and network stack performance. Additionally, the test suite performs various stress tests that monitor the system for memory leaks and performance degradation under loaded conditions. All the tests were executed on the exact same platform we always use — an ordinary PC with an Intel x86 200-MHz MMX processor.

In one of our more sophisticated tests, we generate two simultaneous interrupts (see Listing One) and observe how the system handles them. Both interrupts — one with a higher priority than the other — are generated with less than 100 ns delay between them. This qualifies as simultaneous interrupts. The system services the highest priority interrupt first, followed by its low-priority counterpart. Figure 1 shows the results for the interrupt latency; the time the RTOS needs to service both interrupts. Clearly, the QNX RTOS doesn't experience the slightest difficulty in handling simultaneous interrupts, as it did not even take 6 ms (microseconds) to start servicing the low-priority interrupt in the worst-case scenario we encountered. The high-priority interrupt was always serviced within 3 s.

Again, the QNX RTOS uses virtual memory protection. While this mechanism increases robustness, it is often thought of as detrimental to the system's performance. To assess the performance impact of this memory protection mechanism, we executed our thread switch latency test two different ways. The first test measures the latency to switch between threads belonging to the same process. All threads within a process share the same address space. On the other hand, every process (application, device driver, or whatever) has its own private virtual memory space. That's why we repeated the same test with each thread residing in a different process, measuring the process switch latency. Figure 2 presents the results. As you might expect, switching between processes is slower, but not by much. If you're going to use an RTOS in a mission-critical system, virtual memory protection is a must-have. The performance penalty does not weigh up against the increase in stability and robustness. Only in the smallest and most resource-constrained embedded systems, memory-protection mechanisms and processes may be inappropriate because of the overhead they cause.

Priority inversion can occur when at least three threads are sharing a common resource that is, for example, protected by a mutex. If the RTOS doesn't have the capability to detect and recover from this situation, it's high-priority tasks may be delayed indefinitely. We executed a test that created a priority-inversion situation and verified how long it would take the QNX RTOS to recover (for a detailed explanation of this test, see "Report Definition and Test Plan" available at http://www.dedicated-systems.com/encyc/buyersguide/rtos/eval_roadmap.htm#reportdef). Figure 3 shows the test results.

Again, our test suite includes stress tests. During one test, we expose the system to a constant stream of periodic interrupts, and check if the system is able to service all of them. The purpose of this is to ascertain the maximum interrupt frequency that can be sustained during long periods of time, without losing interrupts. We do this in two steps. During the first phase of the test, the system is exposed to only 1000 interrupts (set to occur at a particular frequency), while we count how many of those interrupts were serviced. Table 1 shows the frequency threshold at which the system inevitably starts losing interrupts. The results gave us a rough estimate of the maximum interrupt frequency the system could handle during short periods of time.

During the second phase, we repeat the same tests, but this time around, we generate 1 billion (10⁹) interrupts, keeping the system busy for several hours. Naturally, the longer the system is exposed to these interrupts, the more likely it is that some of them will not get serviced. This is reflected by the results in Table 2: The system lost one single interrupt at 8 s, and lost more than we were able to detect with our test setup at 7 s. Needless to say, the QNX RTOS performed well in this stress test.

Be aware that we used a minimal ISR; it simply incremented a counter so we could keep track of the number of interrupts that were serviced. You should also keep in mind that this test not only stresses the RTOS, but also the PC hardware. So no hard conclusions can be drawn as to which part of the system is to blame for losing interrupts.

Conclusion

At the outset, I mentioned that the QNX RTOS is regarded as a robust and reliable system. Looking at the results of this evaluation, it is fair to say that this reputation is justified and well deserved. The system never exhibited any signs of instability during our stress tests and always responded in a fast and predictable manner. Finally, the QNX RTOS is downloadable free for noncommercial use at http://get.qnx.com/.

DDJ

Listing One

/****************************************************************************/
/*** This test program installs two interrupt handlers; one hooking into ***/
/*** into IRQ9, the other one into IRQ10.                                ***/
/*** Both interrupts simultaneously generated by two PCI bus exersizers. ***/
/*** This test program measures the time the system needs to react to    ***/
/*** both interrupts. By adding a small delay between the interrupts,    ***/
/*** this program can also be used to verify that the interrupt handling ***/
/*** is prioritized, and that the handlers can be nested.                ***/ 
/****************************************************************************/

/* Include files */
#include <stdio.h>
#include <signal.h>
#include <pthread.h>
#include <sys/neutrino.h>
#include <sched.h>
#include <sys/mman.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <time.h>
#include <semaphore.h>
#include <stdlib.h>
#include "trace.h"

/* Defintion of some constants */
#define PCI_MEMORY   0xE1000000
#define P_DRIVE_1    0xE1100000
#define P_DRIVE_2    0xE1200000
#define PCI_IRQ_1    0x09
#define PCI_IRQ_2    0x0a

/* Global variables */
unsigned long*    pulp_trace;               // PCI_MEMORY
unsigned long*    pulp_pdrive_1;            // P_DRIVE_1
unsigned long*    pulp_pdrive_2;            // P_DRIVE_2
int               iFlag1, iFlag2;

/* Function prototypes */
const struct sigevent*  isr_handler_1(void* pvp_arg,int id);
const struct sigevent*  isr_handler_2(void* pvp_arg,int id);

/***************************************************************************/
/*** The main program entry. This function sets up the interrupt handlers **/
/*** and initializes the hardware.                                        **/
/***************************************************************************/
void main()
{
   int nb_trace,  i_status, i, l, b;
   /* Set this thread to highest priority */
   setprio( 0, 63 );
   /* Map the PCI memory into this thread's virtual address space */
   pulp_trace = (unsigned long*) mmap(0, 4, PROT_READ | 
                 PROT_WRITE,MAP_PHYS | MAP_SHARED, NOFD, PCI_MEMORY);
   /* Map the P_drive_1 into this thread's virtual address space */
   pulp_pdrive_1 = (unsigned long*) mmap( 0, 4, PROT_READ | 
                 PROT_WRITE,MAP_PHYS | MAP_SHARED, NOFD, P_DRIVE_1);
   /* Map the P_drive_2 into this thread's virtual address space */
   pulp_pdrive_2 = (unsigned long*) mmap( 0, 4, PROT_READ | 
                 PROT_WRITE,MAP_PHYS | MAP_SHARED, NOFD, P_DRIVE_2);
   /* Initialise P-Drive-1 */
   *(pulp_pdrive_1 + 0x68/4) = INIT_CODE_PDRIVE;
   *(pulp_pdrive_1 + 0x60/4) = 0x01;

   /* Initialise P-Drive-2 */
   *(pulp_pdrive_2 + 0x68/4) = INIT_CODE_PDRIVE;
   *(pulp_pdrive_2 + 0x60/4) = 0x01;

   /* Enable I/O privilege. You need to have root privileges */
   /* to execute this */
   ThreadCtl(_NTO_TCTL_IO, 0);

   /* Attach the high-priority ISR */
   i_status= InterruptAttach(PCI_IRQ_1, isr_handler_1 , NULL, 0, 0);
   if(i_status==-1) printf(" error attaching ISR-1");

   /* Attach the low-priority ISR */
   i_status= InterruptAttach(PCI_IRQ_2, isr_handler_2 , NULL, 0, 0);
   if(i_status==-1) printf(" error attaching ISR-2");

   /* Enable interrupts */
   InterruptEnable();
   sleep(1);

   /* Generate interrupts */
   for(;;)
   {
      /* Write 1 to PDRive mailbox register 0 to generate interrupt */
      *( pulp_pdrive_1 + ( 0x40/4 ) )= 0x01;    
      *( pulp_pdrive_2 + ( 0x40/4 ) )= 0x01;

      /* Write traces to PCI bus to calculate interrupt latencies   */
      /* Write the only until the interrupts have been serviced.    */ 
      /* This prevents the trace buffer from filling up immediately */
      for(i=0; i<250; i++)
      {
         *pulp_trace = MAIN_LOOP_TRACE;
         if((iFlag1==1)&&(iFlag2==1)) break;
      }
      /* Dummy loop to create a busy delay */
      for(l=0; l<2000; l++)
      {
         i = i << (b+5*3); b++;
      }
      iFlag1=0;
      iFlag2=0;
   }
   exit(0);
}
/***************************************************************************/
/*** The interrupt handler that is hooked to IRQ9                        ***/
/***************************************************************************/
const struct sigevent* isr_handler_1(void* pvp_arg, int id)
{
   /* Write the trace to get interrupt latency and re-init the PDrive */
   *pulp_trace    = ISR1_TRACE;
   *(pulp_pdrive_1 + (0x64/4)) = 0x1;
   iFlag1 = 1;
   return 0; 
}
/***************************************************************************/
/*** The interrupt handler that is hooked to IRQ10                       ***/
/***************************************************************************/
const struct sigevent* isr_handler_2(void* pvp_arg, int id)
{
   /* Write the trace to get interrupt latency and re-init the PDrive */
   *pulp_trace    = ISR2_TRACE;
   *(pulp_pdrive_2 + (0x64/4)) = 0x1;
   iFlag2 = 1;
   return 0; 
}

Back to Article