Strict control over the association of operations, threads, and networking resources is essential for many DRE applications. This column explores Real-time CORBA features that allow DRE applications to select protocol properties and to explicitly bind clients to server objects using pre-allocated connections, priority bands, and private connections.

May 01, 2002
URL:http://www.drdobbs.com/real-time-corba-part-4-protocol-selectio/184403827

Figure 1: Base_Station, Controller, and Drone objects described using CORBA IDL

Figure 2: The server designating the protocols available to the client

Figure 3: Tagged components contained within an object reference

Figure 4: A multithreaded Base_Station server that sends stop, turn, and query_state requests to Drone

Figure 5: _bind_priority_band requests

Figure 6: stop and return using a different connection than query_state

Figure 7: Priority banded, private connections used between client and server

Figure 8: Override order precedence, from highest to lowest

Table 1: Real-time CORBA features

Object Interconnections: Real-time CORBA, Part 4: Protocol Selection and Explicit Binding

Douglas C. Schmidt and Steve Vinoski

Introduction

This is the fourth and final installment in our series of columns describing how CORBA has evolved to support DRE (distributed real-time and embedded) applications. Our previous columns motivated the need for the capabilities offered by Real-time CORBA [1], described the Real-time CORBA policies and mechanisms that enable clients and servers to indicate the relative priorities of their requests so that ORBs can enforce these priorities end-to-end [2], and examined Real-time CORBA's support for thread pools and synchronizers [3]. This column explores Real-time CORBA features that allow DRE applications to select protocol properties and to explicitly bind clients to server objects using pre-allocated connections, priority bands, and private connections.

As in our earlier columns, we'll illustrate key Real-time CORBA capabilities using the planetary mapping system described in [1]. In this system, semi-autonomous drone vehicles are involved in a distributed computation to map a planet's surface. Human operators define high-level exploration goals for the drones via a Base_Station object, which provides set points for Controller objects. Controller objects control the drones remotely using Drone objects, which reside on the drone vehicles. Drone objects expose operations for monitoring and controlling individual drone behavior. Each drone sends information obtained from its sensors back to the Base_Station via its Controller object. The Base_Station, Controller, and Drone objects are all described using CORBA IDL, as outlined in Figure 1.

In this column, we'll apply the Real-time CORBA features shown in Table 1 to address a number of design challenges related to controlling protocol selection and explicit bindings of network resources between client and server objects. The remainder of this column illustrates how to apply the Real-time CORBA features outlined in Table 1 to address key protocol and networking design challenges that arise in our planetary mapping application. Although we describe these design challenges in the context of planetary mapping, they are relevant to many other DRE applications, such as avionics mission computing, telecom call processing, and process control systems. In particular, strict control over the association of operations, threads, and networking resources is essential for many DRE applications. Earlier versions of CORBA provided no standard way for clients or servers to configure their protocol properties or explicitly bind networking resources between clients and server objects. These features are necessary, however, to minimize end-to-end priority inversion, as well as to bound latency and jitter for distributed applications with deterministic real-time QoS (quality of service) requirements.

Selecting and Configuring Protocol Properties in Real-time CORBA

CORBA uses inter-ORB communication mechanisms to exchange requests between clients and servers. These mechanisms are built upon lower level protocols that provide various types of QoS. IOP (Inter-ORB protocol) instances are composed of both an ORB protocol and a mapping to a specific underlying transport protocol. For example, the IIOP (Internet Inter-ORB Protocol) is a mapping of the GIOP (General Inter-ORB Protocol) onto TCP/IP. Thus, an IOP consists of two protocol layers — ORB and transport — each having its own set of protocol properties, such as queueing sizes, packet routing strategies, and latency characteristics.

Unfortunately, the standard Internet protocols based on TCP/IP do not provide end-to-end real-time response. Communication between DRE application objects must therefore often use different protocols, such as VME or Link 16. In particular, some requests exchanged between Drone and Controller objects cannot be delayed due to flow control or queueing. Real-time CORBA therefore defines an interface that permits applications to specify ORB- and transport-specific protocol properties that control various network protocol features, such as ATM virtual circuits or Link 16 dynamic channel allocation settings. Each ORB/transport protocol property is defined by a Protocol structure that ultimately resides in a sequence of structures called a ProtocolList, which is defined by the following CORBA IDL:

interface ProtocolProperties;

struct Protocol { 
 IOP::ProfileId protocol_type;
 ProtocolProperties orb_protocol_properties;
 ProtocolProperties transport_protocol_properties;
};
typedef sequence <Protocol> ProtocolList;

The order in which protocol properties appear in the ProtocolList is significant — it allows applications to indicate the order of their protocol preferences. For example, our planetary mapping application specifies that Link 16 is more preferable than other protocol combinations, such as IIOP.

To allow applications to select and configure their desired ORB/transport protocol properties, Real-time CORBA defines the QoS policies described below:

• Server-side protocol properties. CORBA servers can use the ServerProtocol policy to select which protocol(s) to configure into an object reference. This policy can be passed with other POA policies when the create_POA operation is invoked on the PortableServer::POA interface. The ServerProtocol policy has two purposes: it (1) publishes a list of available protocols to clients and (2) defines protocol configuration attributes for server connections. The POA ensures that the ordering of profiles in object references conforms to the ordering of protocols specified in the ServerProtocol policy. A server can therefore export its protocol preferences to clients by passing them in object references whose profiles are arranged in a particular order. When a client receives the object reference, it can either accept the server's preference or use a different selection criterion.
• Client-side protocol properties. Client applications can use the ClientProtocol policy to select which protocol(s) to use when they connect to objects. This policy is applied when a client obtains a binding to an object. The ClientProtocol policy indicates the protocol properties a client is interested in, as well as the ordering of its preferences. The ClientProtocol policy can be set either by a client or server, but not by both for the same object reference. Servers can publish particular protocol requirements and preferences on a per-object basis. In contrast, clients can use this policy to change protocol policies on a per-invocation basis. If the ClientProtocol policy is set on the server, it's propagated to the client in the object reference, as shown in Figure 2.

Figure 2 illustrates how a server can designate the protocols available to the client. The server publishes the Link16 and VME protocols, in that order, in a tagged component contained within the object reference, which allows a server to enforce specific inter-ORB protocol requirements on its clients. Figure 3 illustrates how tagged components are contained within an object reference.

The client must abide by the ClientProtocol policy propagated by the server and select from one of the protocols designated as tagged components, ignoring any protocols that it does not support. In the case of the planetary mapping system, it selects the Link 16 protocol since that works over a radio link between drones and the base station controller.

The particular properties for specific protocols can be defined via interface inheritance. For example, the standard TCP protocol properties are shown below:

interface TCPProtocolProperties: ProtocolProperties { 
 attribute long send_buffer_size;
 attribute long recv_buffer_size;
 attribute boolean keep_alive;
 attribute boolean dont_route;
 attribute boolean no_delay;
};

This protocol property interface permits applications to set common attributes of TCP endpoints. For example, the send_buffer_size and receive_buffer_size attributes can set the size of endpoint socket queues. Many TCP implementations use these values to determine the TCP window size, which in turn affects end-to-end throughput. If the keep_alive attribute is enabled, TCP will send a probe on inactive connections to verify that they are still valid. Finally, the no_delay attribute disables TCP's Nagle algorithm so that small requests can be sent even if earlier requests have not yet been acknowledged.

It's ironic that the only protocol property interface specified by Real-time CORBA is for TCP/IP, which is not suitable for most DRE applications! The reason for this, of course, is that TCP/IP is such a ubiquitous protocol that it's possible to standardize its protocol properties in a portable interface. Hopefully, the OMG will eventually be able specify standard interfaces for more common real-time protocols, such as VME and Link 16.

For completeness, we'll show how to configure a ProtocolList containing the TCP/IP and (imaginary) Link 16 protocol properties. As usual, we start by using the standard CORBA::ORB_init factory operation to obtain an object reference to the ORB:

CORBA::ORB_var orb = CORBA::ORB_init (argc, argv);

We then use resolve_initial_references and _narrow to obtain an object reference to a Real-time CORBA ORB:

CORBA::Object_var obj =
 orb->resolve_initial_references ("RTORB");
RTCORBA::RTORB_var rt_orb = RTCORBA::RTORB::_narrow (obj);

Using the rt_orb object reference, we create the protocol properties for TCP/IP as follows:

RTCORBA::ProtocolProperties_var tcp_properties =
 rt_orb->create_tcp_protocol_properties (
64 * 1024, /* send buffer */
64 * 1024, /* recv buffer */
false, /* keep alive */
true, /* dont_route */
true  /* no_delay */);

Next, we'll configure the list of protocols to use:

RTCORBA::ProtocolList plist; plist.length (2);
plist[0].protocol_type =
  IOP::LINK16_PROTOCOL_IOP; /* This is non-standard */
plist[0].trans_protocol_props =
 /* Use a proprietary ORB interface for Link 16 */
plist[1].protocol_type =  IOP::TAG_INTERNET_IOP; 
plist[1].trans_protocol_props = tcp_properties;
RTCORBA::ClientProtocolPolicy_var policy =
 rt_orb->create_client_protocol_policy (plist);

The ClientProtocol policy can be applied at the time of binding on the client, or during POA creation on the server. When applied on the server, this policy, together with all the other client-exposed policies, are embedded inside an IOR-tagged component and passed to clients for inspection.

Explicit Binding Mechanisms in Real-time CORBA

Before Real-time CORBA was standardized, the CORBA specification only supported implicit binding. In this model, resources along the path between a client and its server object are established on-demand (e.g., after a client's first invocation on the server). Implicit binding helps preserve location transparency by allowing clients to access remote objects or collocated objects using a common programming model. In addition, it helps conserve OS and networking resources, such as socket handles or ATM virtual circuits, by deferring the creation of network connections until they are actually used. Although not strictly part of the implicit binding model, many ORBs also allow multiple client threads in a process to be multiplexed through shared network connections to their corresponding server.

Unfortunately, implicit binding and connection multiplexing are inadequate for DRE applications with stringent QoS requirements. In particular, deferring connection establishment until run time can increase latency and jitter significantly. Moreover, the use of connection multiplexing can yield substantial priority inversion due to head-of-line blocking associated with connection queues that are processed in FIFO order. Figure 4 illustrates these problems in the context of our planetary mapping system.

Figure 4 shows a multithreaded Base_Station server that sends stop, turn, and query_state requests to a Drone. Although the stop and turn requests run in higher priority threads than the query_state request, an ORB that's configured to use implicit binding and multiplexed connections may end up sending query_state first. Unfortunately, this will cause head-of-line-blocking and priority inversions if the communication channel is processed in FIFO order.

To avoid these problems, the Real-time CORBA specification defines a group of orthogonal explicit binding mechanisms that support the following capabilities:

• A connection pre-establishment mechanism, which eliminates a common source of operation jitter.
• A PriorityBandedConnection policy, where the invocation priority of the client determines which connection is used.
• A PrivateConnection policy, which guarantees non-multiplexed connections.

We discuss each of these policies and mechanisms below and explain how they help to improve the predictability and responsiveness of our planetary mapping application.

Connection pre-establishment. To pre-establish network connections, we use the _validate_connection operation defined in the CORBA::Object interface. This operation enables clients to pre-establish connections to servers and to control how client requests are sent over these connections. For example, the following code illustrates how the Base_Station can pre-establish a connection with a Drone. We start by getting an object reference to a Drone, which could be obtained via a Naming Service or some pre-configured header file:

Drone_var drone = ...; 

We then pre-establish a connection by invoking _validate_connection on the drone object reference:

CORBA::PolicyList_var invalid_policies;
CORBA::Boolean success =
 drone->_validate_connection (invalid_policies);

This operation causes a _bind_priority_band implicit request to be sent to the server. Implicit requests are implemented by the ORB, not by an application object, and are used for internal inter-ORB communication and configuration. When the server receives the _bind_priority_band request, it allocates resources to handle the connection.

Priority banded connections. In addition to pre-established connections, DRE applications also often require their ORB end systems to minimize priority inversions, so that high-priority operations are not queued behind low-priority operations. Real-time CORBA enables applications to use different connections for different priority ranges via the PriorityBandedConnection policy.

Priority-banded connections allow clients to do the following:

• Specify explicit priorities for each network connection.
• Select the appropriate connection at run time based on the CORBA priority of the thread that invoked an operation.

Clients can specify policies that define one or more priority bands when they establish connections explicitly. The policy can also be set by a server at POA creation time, in which case the ORB exposes it to clients by embedding it inside an IOR-tagged component. This policy should be set only on the client or the server, but not on both, or an attempt to bind will fail.

Priority band information is exported to the server within the service context of the first invocation sent across the connection. For instance, this information can be passed in the _bind_priority_band requests shown in Figure 5. The requested priority is passed along in the service context field associated with the _bind_priority_band operation. Subsequent requests on these connections are then processed at the requested priority levels. If a client does not send a _bind_priority_band request, an implicit bind will be performed when the first invocation is sent over the connection. The service context of this request must contain the CORBA priority range (i.e., minimum and maximum values) for the banded connection. The server then allocates any necessary resources to ensure subsequent requests arriving on this connection will be processed at the desired priority.

To illustrate the use of priority-banded connections in our planetary mapping client application, we'll start by creating the priority bands:

RTCORBA::PriorityBands bands (2); bands.length (2);

We then initialize the bands with a range of priorities, such as:

bands[0].low = 0;
bands[0].high = 150;

We can also initialize a "range" of a single priority:

bands[1].low = 200; 
bands[1].high = 200;

We then pass the bands to the create_priority_banded_connection_policy factory method:

CORBA::Policy_var policy =
 rt_orb->create_priority_banded_connection_policy (bands);

As a result of applying this policy to our planetary mapping client, the stop and turn operations will now use a different connection than the query_state operation since they run in different threads at different priorities (see Figure 6).

Private connections. Many ORBs support multiplexed connections, which are designed to conserve connections and other limited OS resources. However, DRE applications often require so-called "private" (i.e., non-multiplexed) connections, which cannot be reused for another two-way request until the reply for the previous request is received. For example, the stop operation in our planetary mapping application should use a dedicated connection to avoid head-of-line blocking with other operations. To support this use case, Real-time CORBA defines a PrivateConnection policy, which allows clients to select private connections that minimize the duration of any end-to-end priority inversions. Oddly, there is no API in Real-time CORBA to explicitly request a multiplexed connection (i.e., this is considered an ORB implementation detail).

Figure 7 illustrates the use of priority banded, private connections between our planetary mapping client and a server.

Each client operation (i.e., stop, turn, and query_state) running in a different thread is sent to the server over a pre-allocated connection that's assigned to a fixed priority range. The server ORB then processes the corresponding servant upcall at the specified priority and sends the reply across the same non-multiplexed connection. This combination of features ensures that end-to-end priorities are maintained and that key sources of priority inversion are eliminated.

We can program the PrivateConnection policy in our client application as follows:

policies[0] = rt_orb->create_private_connection_policy ();
CORBA::Object_var object = drone->_set_policy_overrides
 (policies, CORBA::ADD_OVERRIDES);

The code fragment shown above first creates the PrivateConnection policy using the factory method create_private_connection_policy and then applies the policy on the drone object reference via the _set_policy_overrides operation. This operation overrides the policy at the Object scope. Client policies can be specified at three "overriding levels":

1. ORB-level via CORBA::PolicyManager.
2. Thread-level via CORBA::PolicyCurrent.
3. Object-level via calls to _set_policy_overrides on an object reference, as shown above.

Figure 8 illustrates the order in which these overrides take precedence, from highest to lowest.

Server policies can also be specified at three overriding levels:

1. ORB-level through CORBA::PolicyManager.
2. POA-level passed as arguments to POA::create_POA.
3. Some policies, such as object priority, can be set at the Object-level via operations on the RTPOA.

Concluding Remarks

The Real-time CORBA specification adds powerful QoS capabilities to regular CORBA to improve application predictability by bounding priority inversions and managing system resources end-to-end. The Real-time CORBA standardization effort began in 1997. By the time the Real-time CORBA 1.0 specification was officially integrated into the CORBA 2.4 document [4] in October 2000, implementations of Real-time CORBA ORBs, such as TAO from OCI (<http://theaceorb.com/>), ORBExpress from OIS (<www.ois.com>), and HighComm from Highlander (<www.highcomm.com>), had matured to the point where they were being used in a wide-range of mission-critical DRE systems, such as process control, avionics mission computing, telecommunication call processing, and missile defense. The remarkably rapid success of Real-time CORBA for DRE applications has stemmed from substantial R&D efforts over the past decade that focused on developing patterns and frameworks as a means to promote the development and reuse of successful middleware technology.

• Patterns capture successful solutions to commonly occurring software problems that arise in a particular context [5]. Patterns can simplify the design, construction, and performance tuning of middleware and applications by codifying the accumulated expertise of developers who have confronted similar problems before. Patterns also raise the level of discourse in describing software design and programming activities.
• Frameworks are concrete realizations of groups of related patterns [6]. Well-designed frameworks reify patterns in terms of functionality provided by the middleware itself, as well as functionality provided by an application. A framework also integrates various approaches to problems where there are no a priori, context-independent, optimal solutions. Middleware frameworks [7] can include strategized selection and optimization patterns so that multiple independently developed capabilities can be integrated and configured automatically to meet the functional and QoS requirements of particular applications.

It's important to recognize that the Real-time CORBA specification addresses some — but by no means all — important real-time application development challenges. For example, it focuses primarily on "fixed-priority" real-time applications. In addition to Real-time CORBA 1.0, the OMG has adopted the following related specifications in the past three years:

• Minimum CORBA, which removes non-essential features from the full OMG CORBA specification to reduce footprint so that CORBA can be used in memory-constrained embedded systems.
• CORBA Messaging, which exports additional QoS policies, such as timeouts, request priorities, and queueing disciplines, to applications.
• Fault-tolerant CORBA, which uses entity redundancy of objects to support replication, fault detection, and failure recovery.

Robust implementations of these CORBA capabilities and services are now available from multiple vendors. The OMG is also expanding the scope of open systems to include an even wider range of applications with the advent of emerging standards, such as Dynamic Scheduling Real-Time CORBA [8] and the Real-time Notification Service. Future columns will discuss some of these specifications.

A PowerPoint tutorial version of our columns on the Real-time CORBA specification is available at <www.cs.wustl.edu/~schmidt/RT-CORBA.ppt>. If you have comments, questions, or suggestions regarding CORBA or our column, please let us know at [email protected].

References

[1] D.C. Schmidt and S. Vinoski. "Object Interconnections: Real-time CORBA, Part 1: Overview and Motivation," C/C++ Users Journal C++ Experts Forum, December 2001, <www.cuj.com/experts/1912/vinoski.htm>.

[2] D. Schmidt and S. Vinoski. "Object Interconnections: Real-time CORBA, Part 2: Applications and Priorities," C/C++ Users Journal C++ Experts Forum, January 2002, <www.cuj.com/experts/2001/vinoski/vinoski.htm>.

[3] D. Schmidt and S. Vinoski. "Object Interconnections: Real-time CORBA, Part 3: Thread Pools and Synchronizers," C/C++ Users Journal C++ Experts Forum, March 2002, <www.cuj.com/experts/2003/vinoski.htm>.

[4] Object Management Group. "The Common Object Request Broker: Architecture and Specification Revision 2.4," OMG Technical Document formal/01-02-33, October 2000, <www.omg.org/cgi-bin/doc?formal/01-02-33>.

[5] D. Schmidt, M. Stal, H. Rohnert, and F. Buschmann. Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Objects (Wiley and Sons, 2000), <www.cs.wustl.edu/~schmidt/POSA/>.

[6] D. Schmidt and S. Huston. C++ Network Programming: Systematic Reuse of ACE and Frameworks, (Addison-Wesley, 2002), <www.cs.wustl.edu/~schmidt/ACE/book2>.

[7] Richard E. Schantz and Douglas C. Schmidt. "Middleware for Distributed Systems: Evolving the Common Structure for Network-centric Applications," Encyclopedia of Software Engineering (Wiley and Sons, 2002), <www.cs.wustl.edu/~schmidt/PDF/middleware-chapter.pdf>.

[8] Object Management Group. "Dynamic Scheduling Real-Time CORBA 2.0 Joint Final Submission," OMG Document orbos/01-06-09, April 2001, <http://cgi.omg.org/cgi-bin/doc?orbos/01-06-09.pdf>.

About the Authors

Steve Vinoski is vice president of Platform Technologies and chief architect for IONA Technologies and is also an IONA Fellow. A frequent speaker at technical conferences, he has been giving CORBA tutorials around the globe since 1993. Steve helped put together several important OMG specifications, including CORBA 1.2, 2.0, 2.2, and 2.3; the OMG IDL C++ Language Mapping; the ORB Portability Specification; and the Objects By Value Specification. In 1996, he was a charter member of the OMG Architecture Board. He is currently the chair of the OMG IDL C++ Mapping Revision Task Force. He and Michi Henning are the authors of Advanced CORBA Programming with C++, published in January 1999 by Addison Wesley Longman. Steve is also IONA's primary representative to the W3C (World Wide Web Consortium) Web Services Architecture Working Group.

Doug Schmidt is an associate professor at the University of California, Irvine. His research focuses on patterns, optimization principles, and empirical analyses of object-oriented techniques that facilitate the development of high-performance, real-time distributed object computing middleware on parallel processing platforms running over high-speed networks and embedded system interconnects. He is the lead author of the books Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Objects, published in 2000 by Wiley and Sons, and C++ Network Programming: Mastering Complexity with ACE and Patterns, published in 2002 by Addison-Wesley. He can be contacted at [email protected].

Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.