Channels ▼
RSS

C/C++

Real-Time Systems & RT CORBA

Source Code Accompanies This Article. Download It Now.


December, 2004: Real-Time Systems & RT CORBA

Managing distributed real-time systems with CORBA extensions

William is the Chief Software Engineer for Stage Logic LLC, a small software development company. He specializes in developing CORBA-based real-time systems. He can be contacted at bill@stagelogic.com.


Once thought to be the panacea of distributed computing, CORBA's complex structure often made it impractical in much of the rapid design world of enterprise computing. However, CORBA is far from dead. In fact, it lives on in the world of real-time distributed computing. In the real-time computing arena, CORBA's flexibility makes it ideal for systems where speed and predictability are every bit as important as time to market.

The ability to use CORBA for real-time applications is provided by the Real-Time CORBA Specifications (http://www.omg.org/technology/documents/formal/real-time_CORBA.htm). The RT CORBA spec is a set of CORBA extensions that lets systems with real-time scheduling requirements meet those requirements across multiple CORBA objects, possibly residing on different object-request brokers (ORBs).

The RT CORBA extensions reside on top of the operating system. The only interaction that an RT CORBA client or servant need have with the underlying operating systems' real-time mechanisms is through the use of priority mappings, which map cross-platform CORBA priorities to local OS-level priorities. This level of disconnect lets the RT CORBA ORB have no real-time OS constraints, which would require an RTOS that provided certain features. Because it has no such requirements, RT CORBA is not limited to real-time operating systems. On a nonreal-time OS, you lose the hard real-time predictability provided by the RTOS but still benefit from the prioritized scheduling features of RT CORBA, which may be sufficient for many soft real-time applications.

Priority Mapping

Priority mappings are the most important basic concept in RT CORBA. Priority mappings let individual CORBA components specify their priority using cross-platform CORBA priorities, which range from 0 to 32767. These CORBA priorities are then mapped to OS native priorities using a definable priority mapping.

For example, consider a system for routing sensor data to processors that handle the data. I have a number of different sensor processes, each running on a different embedded device, which receives a stream of data from input equipment. The different sensors process a myriad of different types of data, each of which has a different importance to the system. Additionally, specific data can affect the importance of a sensor at any given time. The sensors send their data to a central router, which analyzes incoming data and routes it to one or more data sinks for processing.

As data comes in to the sensors, they do some rudimentary processing to analyze the priority of the data at that point in time, and categorize it into one of four importance levels: low, medium, high, and emergency. These levels could simply have been mapped to the CORBA priorities 0 through 3, but I want room to expand to more priority levels later on down the road. Consequently, I (semiarbitrarily) chose to map them to 1024, 2048, 4096, and 8192.

Now that I have my priority levels chosen, it might be nice to have them mapped to native priority levels on the system. While 32,000-plus priority levels are fun, they're not practical on most platforms. I accomplish this feat by extending the RTCORBA::PriorityMapping class and write to_native() and to_corba() methods (see the SensorPriorityMapping class in Listing One). For robustness, I also extend the mapping to use ranges that encompass my four levels earlier. That way, if I later extend my list to encompass more levels, I won't risk crashing any older devices in the system.

How do I start using this new priority mapping? That's a good question. The RT CORBA specification leaves out any standardized method for installing the mapping. Instead, the spec leaves it up to individual ORBs to decide. In my case, I was using TAO (The ACE ORB; http://www.cs.wustl.edu/~schmidt/TAO.html) as my ORB implementation, and this was done through a PriorityMappingManager. Your ORB of choice will almost certainly do it differently (unless, of course, your ORB of choice is TAO).

Again, the sensors in my system prioritize the data that they receive into four categories. Once a sensor has performed its analysis and made the categorization, it adjusts its own thread priority to match the priority for the data. The priority for a thread in a CORBA client or server is held in an RTCORBA::Current object. To change its priority, a thread simply has to set the base_priority attribute in the Current object and a native priority change is immediately executed, using the previously installed priority mapping. Example 1 shows that my sensor setting is priority based on the data analysis, prior to sending that data to the router.

Priority Propagation

Priority inversions are bad. Inversions occur when a thread of the execution's priority is implicitly (and unintentionally) reduced by a dependency on a lower priority thread. For example, say, I have Thread A running at priority 50, Thread B running at priority 40, and Thread C running at priority 30. If Thread A requests data from Thread C, it blocks until Thread C returns that data. Thread C, however, won't get to run until Thread B finishes its own processing. Thread B has now implicitly gained a higher priority than Thread A, which is almost certainly not desired.

To avoid priority inversions in my sensor system, I want the threads in the router and data processors that handle data from the sensors to increase their own priority to match the sensor's priority, if they were previously lower. Fortunately, RT CORBA provides a mechanism for just such a propagation model, the "client propagated priority model."

The propagation model used is set on a per-POA basis, by passing a priority model policy when the POA is created. (The Portable Object Adapter, or POA, manages server-side resources for scalability.) The policy to use is created by a call to create_priority_model_policy() on the RTORB, which is obtained by resolving the initial reference "RTORB." The name of the policy to be set for the objects in my sensor system is CORBA::CLIENT_PROPAGATED, which means exactly what it sounds like. When a client makes a call to a CORBA object, that object's servant adjusts its priority upward to meet the client's priority, if necessary. Example 2 shows how I create a POA for the sensor router, with the client propagated priority policy. The initCORBA() and activateRouter() functions in Listing Two provide a more complete example showing the activation of the SensorRouter object.

The other priority model available in RT CORBA is SERVER_DECLARED, which means that the server has a set priority that it runs at, no matter what the client's priority is. Clients can, however, query the server's declared priority. The server declared model can be useful for objects, such as a video controller, that need to run at a constant priority. The priority that a server-declared object is running at is distributed within that object's reference, and can be obtained by retrieving the PriorityModelPolicy from the reference using _get_policy().

Threadpools

To this point, I've mostly been discussing the actual sensors in a sensor processing system. That makes sense, because there are a lot of them. A lot of sensors means a lot of data, and all that data has to go through my router, which determines which data sinks to send the data to for processing. Simple enough, but a few minutes thought should make it obvious that my router is going to quickly become a bottleneck in the system. Things might be better if I fully distributed things and took it out of the mix; just have the sensors send data directly to the appropriate data sinks.

Unfortunately, sensors sending data directly to sinks doesn't work. A full picture of what data is flowing is necessary to make decisions about which sinks to send data to, and that kind of knowledge and processing is far too data and processor intensive for my sensor devices. That means that I need to make sure my router can efficiently handle all of the data being sent its way.

The naive approach to handling the data would be for the router to deal with things on a first in, first out (FIFO) basis. This can lead to priority inversions, though, if a high-priority piece of data ends up waiting on a low-priority piece that got there first. The solution, of course, is to make the router multithreaded. Just adding more threads doesn't solve the problem by itself, though. If I have 100 threads, I can still get a priority inversion if high-priority data ends up waiting for the 100 low-priority data streams that got there first. Fortunately, RT CORBA provides a solution to this problem, too, in the form of threadpools and threadpools with lanes.

Threadpools let you specify static and dynamic threads on an ORB-wide or per-POA basis. When a POA is created, it creates a pool of static threads, which are kept around to process requests. The dynamic threads, on the other hand, are only created when the static threads are all busy and an additional request is received. When they are done processing the request, they go away.

Threadpool lanes let you create pools of different priority threads. When a request is received, it is filtered into the appropriate threadpool lane and executed. If desired, lanes can also let lower priority lanes be borrowed to run at higher priority if a higher priority request is received and there are no high-priority lanes available. Lanes can also be configured to buffer requests received if there are no lanes available to process a request.

In my sensor processing router, I chose to create four lanes of threadpools—one for each of my four priorities (low, medium, high, and emergency). For each lane, I assign a fixed number of static threads, which will be created when the router is instantiated. Additionally, for each lane, I set an upper limit on the number of dynamic threads it can use, which are created on demand by the ORB.

To construct the lanes, I create an RTCORBA::ThreadpoolLane for each lane, and fill out the appropriate data. These are then compiled into an RTCORBA::ThreadpoolLanes sequence, and passed to create_threadpool_with_lanes(), which is invoked on the real-time ORB; see Example 3. Create_threadpool_with_lanes() returns an RTCORBA::ThreadpoolId, which can be used to associate the threadpool with a POA by setting it as a policy. It can also be used to override the policy of an existing object reference, as in Example 4.

Invocation Timeouts

Once the data reaches the router, it needs to be sent to one or more data sinks to be processed. The processing, though, is time sensitive. If the data hasn't been properly processed within a fixed amount of time, it's better to just drop it and move on to the next piece of data (assuming the priority is less than emergency level). Since the delay could be due to the method call to send the data waiting in a buffer queue for too long, it would be insufficient to just time the duration of the method's processing, internal to the method. Therefore, I need to set an invocation timeout, to constrain the maximum time that the call takes.

Setting a timeout on an invocation is done through the Messaging::RelativeRoundtripTimeoutPolicy interface, which is actually a standard CORBA-specified interface and not specific to RT CORBA. I want to apply this timeout to almost every call that the router makes, so I set the policy for the entire ORB by invoking set_policy_overrides() on the PolicyManager; see Example 5.

As mentioned, though, when an "emergency" priority data packet comes from a sensor, it must get processed, so the timeout needs to be turned off when that data is sent to the sinks. To do that, I make a call to _set_policy_overrides() on my reference to the sink, to get a reference with the overridden timeout policy. In the case of emergency data, predictability is also important, so to avoid any potential delays, I create these policy overridden object references for the data sinks when the sink is registered with the router. It then becomes a simple matter of checking the priority that I have set, and making the call on the appropriate reference.

Mutex

In my router, I have a fixed number of data types, each of which can be sent from multiple sensors. Once the router has received data of a given type, it needs to route it to one or more data sinks. It is important, however, that data of a given type be strictly ordered across the sinks. If Sensor A sends a type of data, and Sensor B sends the same type of data, bad things could happen if one sink gets data from A then B, and another receives B before A.

To ensure the ordering of my sensor data happens appropriately, I take advantage of RT CORBA's mutex feature. The RTCORBA::Mutex interface lets me create a mutex that locks processing of data types across different router threads. Each thread simply acquires a lock on the mutex before processing received data, then releases it when it's done, as in Example 6. The Mutex object uses the same priority inheritance model as the ORB that creates it, so in this case, it will inherit the client propagated model. This gives higher priority data streams priority when acquiring a lock, which prevents the dreaded inversion. I create a new Mutex by calling rtorb->create_mutex().

Conclusion

Overall, the RT CORBA Standard is fairly small—the formal specification is less than 100 pages. However, it's enough to make CORBA useful in distributed real-time systems. Experienced developers know that real-time systems are hard. Scheduling predictability is not an easy task to manage. Scheduling predictability across a network is a nightmare waiting to happen. Real-time CORBA provides a framework that takes the grunt work out of the process and lets you focus on the actual logic of your system.

DDJ



Listing One

class SensorPriorityMapping : public RTCORBA::PriorityMapping
{
  public:
    virtual
    CORBA::Boolean to_native(RTCORBA::Priority corba_priority,
                             RTCORBA::NativePriority& native_priority);
    virtual
    CORBA::Boolean to_CORBA(RTCORBA::NativePriority native_priority,
                            RTCORBA::Priority& corba_priority);
};
CORBA::Boolean
SensorPriorityMapping::to_native(RTCORBA::Priority corba_priority,
                                 RTCORBA::NativePriority& native_priority)
{
  if(corba_priority <= 1024) {
    native_priority = 32;
  }
  else if(corba_priority <= 2048) {
    native_priority = 64;
  }
  else if(corba_priority <= 4096) {
    native_priority = 128;
  }
  else {
    native_priority = 192;
  }
  return true;
}
CORBA::Boolean
SensorPriorityMapping::to_corba(RTCORBA::NativePriority native_priority,
                                RTCORBA::Priority& corba_priority)
{
  if(native_priority <= 32) {
    corba_priority = 1024;
  }
  else if(native_priority <= 64) {
    corba_priority = 2048;
  }
  else if(native_priority <= 128) {
    corba_priority = 4096;
  }
  else {
    corba_priority = 8192;
  }
  return true;
}
Back to article


Listing Two
#define LOW_PRIORITY       1024
#define MEDIUM_PRIORITY    2048
#define HIGH_PRIORITY      4096
#define EMERGENCY_PRIORITY 8192

CORBA::ORB_var orb;
RTCORBA::RTORB_var rtorb;

SensorRouterImpl* sensorRtrServant;
SensorRouter_var sensorRouter;

void initCORBA(int argc, char** argv)
{
  // Initialize the ORB, and assign it to the CORBA::ORB reference orb
  orb = CORBA::ORB_init(argc, argv);
  
  // Resolve the real-time ORB, and assign it to the
  // RTCORBA::RTORB reference rtorb
  CORBA::Object_var object = orb->resolve_initial_references("RTORB");
  rtorb = RTCORBA::RTORB::_narrow(object);
}
void activateRouter(void)
{
  // Create a list of policies to assign to the POA
  CORBA::PolicyList policies(2);
  policies.length(2);
  // Create the propagation policy
  policies[0] = 
    _rtorb->create_priority_model_policy(RTCORBA::CLIENT_PROPAGATED, 
                                         MEDIUM_PRIORITY);
  // Create threadpool lanes for each priority level
  // Values are (in order): 
  //   Default priority, # of static threads, # of dynamic threads
  RTCORBA::ThreadpoolLane lowPriorityLane = {LOW_PRIORITY, 20, 20};
  RTCORBA::ThreadpoolLane mediumPriorityLane = {MEDIUM_PRIORITY, 20, 30};
  RTCORBA::ThreadpoolLane highPriorityLane = {HIGH_PRIORITY, 10, 10};
  RTCORBA::ThreadpoolLane emergencyPriorityLane = 
    {EMERGENCY_PRIORITY,  2, 5};
  RTCORBA::ThreadpoolLanes lanes(4);
  lanes.length(4);
  lanes[0] = lowPriorityLane;
  lanes[1] = mediumPriorityLane;
  lanes[2] = highPriorityLane;
  lanes[3] = emergencyPriorityLane;
  // Create the threadpool, with the no default stacksize, 
  // thread borrowing, or request buffering
  RTCORBA::ThreadpoolId tpoolId =
    rtorb->create_threadpool_with_lanes(0, lanes, false, false, 0, 0);
  // Create the threadpool policy to associate with the POA
  RTCORBA::ThreadpoolPolicy_var tpoolPolicy =
    rtorb->create_threadpool_policy(tpoolId);
  policies[1] = tpoolPolicy;
  // Create POA to use for creating router, using the above policies
  PortableServer::POA_var routerPOA = _rootPOA->create_POA("RouterPOA", 
                            PortableServer::POAManager::_nil(), policies);
  // Create an instance of the servant class
  sensorRtrServant = new SensorRouterImpl;
  // Activate the servant with the POA
  CORBA::Object_var object = routerPOA->activate_object(sensorRtrServant);
  sensorRouter = SensorRouter::_narrow(object);
}
Back to article


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video