Bill is president of Valux (http://valux.com/wj/) and coauthor of the 386BSD operating system. He served as advisor to TeleMuse Networks on web services and architecture. He can be contacted at [email protected].
Web services are often described in terms of their broad consumer reach. For example, web services make it possible for consumers to use wireless PDAs to find books (or even stores that sell a particular book), locate restaurants, and identify events on-the-fly. They also let you create expense reports that can be sent via cellphone to accounting. However, web services can be used even more effectively within datacenter environments.
For instance, one company I worked with required a high-availability IT web-service architecture on which to deploy its web-service-based tools to datacenter technicians and managers. For the tool to be useful, the staff had to be able to rely on the web service with greater than 99.999 percent uptime of the servers, feeds, switches, routers, and storage they maintain. However, current web-service infrastructures are designed with low-cost, high-volume consumer gadgets (such as PDAs and cellphones) in mind, not high availability. Users who have tried to download a complex web page to a PDA may end up reloading the page half-a-dozen times before a complete page is received. Consumers conditioned to receive poor service in exchange for low cost currently don't consider this a problem. In contrast, this isn't good enough in datacenter environments that require accurate real-time information.
Fortunately, consumer products are versatile enough that high-availability web-service architecture extensions can be added that leverage:
- Existing web-services protocols to expose failure/recovery state.
- Middleware systems to extend fault model.
- Network management standards to incorporate into existing industry-wide management platforms.
In this article, I examine a specific case of using extensions to existing web-service protocols. These extensions let you detect client communication errors and transparently proxy in the core of the Internet datacenter (IDC) access.
Background
Unburdened by the need to assemble and maintain application state, thin clients have rapid start-up times and can display real-time data from distributed sources. Yet the price paid is in the dependency on the network transport to get to the server, and the server to perform. Complicated protocols or caching cannot recover the time lost due to network outage.
To manage thousands of servers via PDAs, for example, dozens of requests must be made to find the location, appearance, type, wiring, connectivity, availability, usage, power consumption, and heat dissipation as the display is updated. Real-time perception lets users connect cause/effect of remote actions so much that users need not be present to "feel" how well the machines are working. Most IT support and servicing functions are of the form of "adjust and observe."
Datacenter employees can get information on status from many sources simultaneously—their ISP, colocation provider, network consultant, vendor, and the like. Web services let engineers lightly leverage the sources without having to provide resources.
This is not monitoring, however. Monitoring, whether using SNMP agents or agentless/ad hoc methods, involves the trend and status of equipment. So while elements of an existing monitoring resource may be used, it is extended in a form usable at a remote location. This is the first step in making a virtualized datacenter, which, in turn, becomes an adaptive datacenter.
Monitoring a datacenter differs from that of fault monitoring of cellphones or PDAs. In the former, you must manage a small number of complex and expensive resources (switches, bandwidth) with complex failure modes. In the latter, you have a large number of simple and inexpensive resources (cellphones, PDAs) with fewer—but more difficult to remedy—failure modes.
The Problem
Because of these issues, web services as deployed by consumer-service providers for professional services would be too "flaky" for IDC users, making them useless. If web services are located in different datacenters (see Figure 1), with wireless devices attaching to varying service providers in the Internet "cloud," the problem becomes manifest. While the service availability path between IDCs is usually very high (99.9 to 99.999 percent), a lower service availability (around 99 percent) of ISP connections is seen by the mobile device transiting through the cloud to each datacenter.
While this lower availability is acceptable for web serving and e-mail (where the application depends on only 1-2 paths through the cloud), the web service may require 4-10 paths at a time, which may vary depending on the application's parameters. Service reliability depends on each service getting through to the device on separate paths simultaneously, and it degenerates as the number of paths is increased. (With IT web services, the limitation of the number of simultaneous connections itself limits the number of concurrent information sources that can be tracked in real time.)
In Figure 1, for example, five web services are spread among three different IDC sites. If paths from the Internet device to the IDC via the Internet cloud each had a 99 percent availability, the device would have a 95 percent chance to present the results of the request—one in 20 attempts would fail. Retries don't increase the likelihood of success, since different incomplete sets may arrive at any time. (Because the application requires real-time data, you can't use the prior set of results to fill in the gap. Worse, with IT tools, you need accurate time references of sampled results.)
As illustrated in Figure 2, engineers at Telemuse found that there were no significant losses in normal operational availability with the services in each of the datacenters, and monitoring rarely found drops. This is no surprise, since datacenters use high-quality connectivity (99.9 percent link availability and duplex feeds), high-quality peerage for path diversity, router technology for path fail-over, and professional staff to manage this base in real time. This advantage doesn't extend through the Internet cloud.
Overstressed single points of failure increase as you move away from the core toward the edge. Yet the edge is not totally devoid of path diversity. Frequently, ISPs understand that they can have failure points and provide some redundancy in their networks; however, it is far less than at the core or in an IDC.
Nevertheless, this does not provide path diversity, as you have access through all the IDCs through the cloud directly. Since the web-service gadget does not contain enough resources to intelligently explore multiple paths, the presumption is that all the direct paths to the services constitute primary paths that have a likelihood of working independent of the other paths.
Other Approaches
One approach commonly applied to deal with the separate service problem is to duplicate all services along each connection path. For example, all services would exist in each datacenter. For a small set of proprietary services, this is acceptable and commonplace.
With web services, however, the advantage intended is to economically scale by leveraging ad hoc services throughout the infrastructure. Thus, in one datacenter, you may attach to cage-related services done by different firms with differing implementations. A web service is intended to intermediate between these separate services (with, for example, patch panel codes for wire/cable identification) at different locations and make them appear as "one." They are not unified at the server level. Instead, they are unified at the client level.
The other approach currently in use is to do redundant links of all services through all paths. Thus, a TCP port on each server may be reflected through the other servers through a simple daemon. For example, assume five services. Through each service, you would require five indirect service references, a total of 25 connections, which is the square of the number of services desired.
Again, this is an adequate solution for a small number of services. However, for IT web services, the number of ports is already limited by the complexity of the service provided. Each of these services provides real-time information from different perspectives—the more perspectives offered, the more valuable the service. To fall back to this second approach would result in less diversity of services offered—a prospect considered less valuable to both business and technical customers.
Solution
Web-service gadgets presume all direct paths to the services constitute primary paths. As such, at least one of the web service's connections was always available, and that service could get to the others. Telemuse engineers observed that if any of these paths function, they provide access to a backup path to the other services proxied by an IDC host via the core; see Figure 3.
Web-service software used by all services employs (embedded in its class file) the backup path mechanism. By belonging to the service group (employing the class file), the negotiation between the wireless gadget and the service it uses automatically handles the fail-over from primary to backup communications mechanism as a specialized form of high-availability clustering.
The effect of using a backup path is the means through which embedded monitoring indicates that the service availability has been lost on a member, and the package records group availability as well as how the group availability is broken down by path. In this way, monitoring by a network operation center can show the service quality of a customer and the service quality of the constituent services independently.
While the wealth of datacenter resources at the core cannot be directly leveraged at the edge (as the decision occurs after the edge routes the datagram), it is possible for the web service to leverage its own limited connections as a form of path diversity to get to any datacenter. From this point onward, the datacenter proxies communications with all others.
Instead of requiring that all connections to servers must work for any of the services to work, any connection to a server would allow all services to work; for example, joining the availability of paths set instead of intersecting the availability of paths set (see Table 1).
Practice and Results
To test this approach in practice, we observed that a web-service client always embeds the connection table in any request to any server as part of the high-availability class. We enhanced this by ensuring that all servers proceed to engage a backup path to each server as a proxy for the client; see Figure 4.
If the client's request to any server times out past smoothed round-trip time, the client sends a message on a backup path via an embedded message to a different server. If the message isn't responded to in the accumulated round-trip time, yet another backup path is tried, until all are exhausted and a hard failure is indicated. The same process occurs with information outbound from the server to the client. This is embedded monitoring.
HiAvail.class (see Figure 5) uses the client's connection table as a status report on why it needed to use a backup path. In the server's instance of this class, table entries of dead primary paths are passed to the monitoring center to signal the client primary path failures.
In effect, the client has four to 10 chances of avoiding a loss of data integrity due to communications failure. If the failure is due to network congestion or another transient outage, the recovery time of a client/server TCP connection may greatly exceed the actual outage time (the congestion back-off recovery time). Even if some of the paths go through the same congestion point, the chance of a message making it through is at least doubled. This is a significant improvement over current approaches.
Figure 6 illustrates the most basic use. The dots are the three network hosts and the primary communication session. Surrounding each host is a ring that represents the backup path session, which is a union of all connections present. If a primary path is broken, the request takes successive backup paths to arrive at the destination, and retains the same strategy for a response. A sequence number is used to prevent multiple replays of the same request.
Three cases of node roles are present:
- Requesters get the message to a responder or forwarders.
- Forwarders pass messages between responders and requesters.
- Responders accept messages and commit transactions only when a path to the responder is certain and acknowledged.
Listing One (available electronically; see "Resource Center," page 5) illustrates messaging using primary and backup paths. Listing Two is an outbound request from a PDA, with two requests combined (these could have been separate requests; see responses). Received by the directly connected host, HiAvail.class extracts a request for itself (primaryPath), a request for another server (backupPath), and the status of connections of the client to pass to the monitoring center. The reason for the backup request can be discovered from the missing response implied by the lastResponded time indication, and the implied round-trip timeout of 32 milliseconds. The message to host dcs was overdue and presumed lost by the impatient PDA. The backup message is reencapsulated by HiAvail.class without the primaryPath request and sent to its final destination or dropped if not possible. The primaryPath message is delivered to the relevant service, in this case, XML-RPC; a response is obtained from the service, and a response message returned to the PDA.
Listing Two (also available electronically) shows the first response returned. The PDA extracts the message, again with HiAvail.class, and updates the round-trip time of the primary connection, and discovers the success/failure of the second attempt for the PDA to communicate with dcs indirectly through host frank. If unsuccessful, or if frank is also lost, the PDA moves on to ernest, the least recently used machine. In this case, the backup request has made it out of frank's hands. Ultimately, results are returned to the client application that initiated the primary path request to host frank.
Listing Three (also available electronically) shows the indirect response case. The backup request has bypassed the communications blockage separating the PDA from dcs, the message has been detected as being destined for the XML-RPC service on dcs, a backupPath response has been generated and passed to frank. The symmetry of the timestamps (all relative to the time of the PDA requester) let the monitoring tools of the servers identify where the problem is. In this case, it's on the route between dcs and the PDA, and the service architecture supporting the services is aware of it while PDA users don't need to be—which is the point.
One concern with this approach is the increased load placed on both network and server. This issue revolves around congestion—when communication failures occur (timeouts), do additional communications provoke congestion on an already congested system? The web-service device doesn't have the capacity to indicate which paths are congested and which are not.
Web-service requests outbound from the client device generally are small, and the congestion results from the inbound path to the device. The potential for adding to the congestion problem on uplink is therefore small. Also, because the number of backup link attempts is already limited by the number of connections present, there is no perpetual congestion event created. At most, only N backup messages on M links can occur.
Furthermore, the technique has the potential for reducing congestion upon success with a backup link. Like MPLS, the backup link's selection lets traffic that might continue to congest a path flow by an uncongested path indefinitely. The client-to-server traffic bypasses the congested area of the network.
Conclusion
We recommend embedded monitoring and path diversity for high-availability, reliable web services for professional uses. Other industries that can apply this new network architecture include manufacturing, healthcare, retail, and finance.
Vinton Cerf, cocreator of TCP/IP, has postulated that billions of Internet devices will be connected to the Internet in the future, all sending and receiving requests. It is likely that a considerable percentage will demand reliable service—far more than today. How do we make this scalable and manageable while still retaining reliability and high availability as required?
The answer may lie in experiments with grid computing and the "autonomic computing" initiative as part of an attempt to "virtualize" the datacenter. Adaptive datacenters will deal with this million-device problem in scalability, requiring more automation of the low-level functions, while focusing on higher level intelligent functions.
DDJ
