Lynn is a software architect for Novell. He can be contacted at [email protected].
One of the gaps in the Internet is the lack of programmability for the World Wide Web. HTTP and HTML flatten the world into a uniform view of information, but only people using a browser see the data. We can read web pages with our eyes, but we don't have a similar, simple mechanism for programs to fetch and read the same live data. To reuse parts of web pages within custom applications, we must subject ourselves to the particulars of HTTP and HTML.
Since the Web is often a front end for sophisticated databases and computer processes, we could capitalize on the program integration opportunity this affords. In this article, I'll discuss WIDL, a proposed system for filling this gap, and the Web Automation Toolkit from WebMethods, which supports WIDL.
WIDL as CORBA for the Web
WIDL, short for "Web Interface Definition Language," is a proposal before the W3C. (The WIDL specification is available at http://www.w3.org/Submission/1997/15/Overview.html.) Developed by WebMethods (http://www.webmethods.com/), WIDL is an XML file format for describing programmatic interfaces to the Web. Borrowing from the object community's ORB ideas, WIDL describes an abstract interface to a "service" existing on the Web. The service is described in an interface definition language (IDL).
Through WIDL, the Web can be viewed as a set of network services instead of a set of pages. Using IDL interfaces, WIDL transforms the human view of the Web into an API invocation. From the perspective of IDL consumers, variations in web page implementation and formatting are irrelevant -- the differences are hidden by the IDL. That the Web is driven by HTTP is similarly unimportant. The consuming program sees "natural" objects and methods. HTTP is merely an implementation detail of the IDL. The goal is for computing machinery to drive and consume the Web as if it were a set of distributed computing services.
Stock-Price Example
For the purpose of illustration, I'll build a program (available electronically; see "Resource Center," page 3) that retrieves stock prices from the Web. To complicate matters, I won't build HTTP or HTML parsing into the program, nor will I hardcode stock-related URL locations. Furthermore, I'll add the requirement that the location of the stock information be a matter of system configuration and not something typed into a preference box by an end user.
To solve this kind of problem with WIDL, I'll define and implement a stock-price service. The client program uses this service to retrieve arbitrary stock prices. I start by generating a WIDL service definition. In WIDL, you declare the service in XML using something like Listing One (at the end of this article). This declaration defines a WIDL service arbitrarily named StockPriceService. The preamble identifies this text as XML Version 1.0 and declares that we are using the WIDL DTD Version 2.0 for the schema. This XML text is saved in a file called Quote.widl.
As part of the service declaration, I must identify the web resource involved in implementing the service. I use http://quote.yahoo.com/q, a form-submission page at Yahoo.com where I can retrieve stock prices. Along with the URL, I indicate that I am using an HTTP GET operation to retrieve the stock data.
Bindings
The next step in defining a WIDL service is to fill in the particulars about the URL I am using. In the case of http://quote.yahoo.com/q, I haven't identified what HTTP form variables need to be submitted, what their names are, or what values they should be set to. Nor have I identified what I want to do with the HTML to be returned from the URL when the HTTP GET command is issued. In WIDL, we describe all of this information using a WIDL XML tag named <BINDING>.
WIDL uses a binding to accomplish the illusion of presenting a web resource as if it were an API call. Consumers of the stock-price retrieval service want to view the stock data as an API call with standard input and output values. The problem, of course, is that the stock price isn't really accessible in this fashion -- it's on an HTML page driven from a CGI invocation via a URL. A WIDL binding defines the mapping between the API style input/output variables and the necessary HTTP/HTML operations.
In the case of the stock-price service, I've defined two bindings -- input bindings are identified as StockPriceServiceInput and the output bindings are StockPriceServiceOutput. The input binding is where a consumer of the API will supply the stock symbol to query. The output binding is simply the returned stock price. In both bindings, I'll use WIDL to map a programmatic variable name into an HTTP/HTML concept.
For the input binding, first note that the URL I am using requires two CGI variables to be included as part of the URL GET operation. The designers of the Yahoo web page mandate the names of these two variables: S, the stock symbol name; and D, the desired level of detail returned. I provide alias names for these items so that the caller of the service sees something more descriptive. As Listing Two shows, I then declare the input binding as the WIDL tags. These tags determine that the HTTP request uses form variables named S and D which you programmatically refer to as StockSymbol and DesiredDetail. Furthermore, I supply a value for DesiredDetail but not for StockSymbol. The net effect for WIDL is that only StockSymbol will show up in the resulting service API since DesiredDetail is hardcoded with a value.
The declaration of this input binding is all that is required to fetch the page containing the stock value. This isn't enough, however, to fully automate the web page. I'm still missing two crucial pieces of information. First, I haven't provided any indication of which part of the resulting web page contains the stock prices. All I really have is the entire HTML for the page. Secondly, I haven't specified how the stock price should be returned to the consumer of the service. I don't want to hand unparsed HTML to the program(s) using the service. To address these deficiencies, I need to specify a WIDL output binding for the service.
WIDL output bindings are similar to input bindings in that a programmatic variable name is mapped into a web entity. That is, we provide a meaningful symbolic name that represents some "thing" on the Web. The callers of the service will see the results of the service call in terms of this meaningful name. Nevertheless, the output binding differs from an input binding in several respects. Most of these differences relate to the fact that the output binding is generally concerned with the HTML content of a page rather than the HTTP addressing of that page. In the stock-price service, for example, the output is a stock price that is originally buried within the HTML content.
To facilitate a reasonable output binding, WIDL uses an object model to identify elements of a page. This lets the WIDL service writer use flexible, structured techniques for identifying which parts of the page are the actual values to be returned from the service. The model proposed by WIDL is (at this writing) a variation of the JavaScript object model. Other similar HTML object models exist such as the Document Object Model (DOM) proposal at the W3C. Undoubtedly, should the W3C adopt WIDL, a single, consistent object model will be adopted. Caveat emptor to anyone using an early incarnation of a potential future standard. For the stock-price service presented here, I'll define the output binding as in Listing Three.
This set of tags identifies the output binding named StockPriceServiceOutput. For the stock-price service, I declare a single output variable named StockPrice. Included in the declaration of StockPrice is REFERENCE="doc.html[0].body[0].p[3].table[0].tr[1].td[2].text," which is the WIDL object model specification for the HTML element containing the stock price. The syntax looks familiar if you have written JavaScript. The syntax identifies a table within the HTML code, selecting the third data cell from the second row of the table. That cell value is bound by the <VARIABLE> tag to the symbolic name of StockPrice. When StockPriceService consumers ask for a stock quote, WIDL drills into the HTML content using the object model specification, parses the stock price, and returns the result as a string.
Pattern Matching
Despite the flexibility I gain from WIDL's REFERENCE tag, I also want WIDL to be reasonably "intelligent" about the way it extracts data from HTML pages. You don't want to have to run to a text editor and patch the WIDL file whenever Yahoo.com makes a minor formatting change. The WIDL specification addresses this requirement through the use of some variations on input/output bindings.
The most common binding variation I need to consider is the use of pattern matching in the HTML object model defined by WIDL. This extension lets me name HTML elements using a combination of the JavaScript object model and textual pattern matching. So far, StockPriceService names the third cell in the second row of the first table in the fourth HTML paragraph as the element that contains the stock value. This is risky, because there is no guarantee that the table will remain at this location in the HTML document. To adapt for variations in table placement, change the reference for the output variable to: REFERENCE="doc.html[0].body[0].p[3].table['*Last Trade*'].tr[1].td[2].text." The key difference here is the index of the table[] element. Instead of passing a hardcoded value of zero, I pass a regular expression as the index value. The new REFERENCE looks within the fourth paragraph for a table containing the words "Last Trade." Once such a table is found, I retrieve the appropriate row and cell values. I can further enhance the reference by shortening it to REFERENCE="doc.table['*Last Trade*'].tr[1].td[2].text," which searches the entire HTML (not just the fourth paragraph) for the appropriate table. The pattern matching significantly increases the reliability of the WIDL definitions. Variations in HTML structure are not nearly as likely to break our WIDL interfaces.
In some situations, the pattern matching still is not strong enough. We may need to authenticate to a web server, parse variable page structures, chain together pages based on page content, build URL addresses from web page content, specify failure conditions, or parse non-HTML files. WIDL can accommodate each of these through additional WIDL tags. I'll use one of these more sophisticated options for the StockPriceService example.
To round out the service, I include a back-up location on the Web where you can retrieve stock prices, in the event http://quote.yahoo.com/ fails. All of this is transparent to consumers of the service. I do this by defining a second stock price service named StockPriceService2; see Listing Four.
StockPriceService2 is nearly identical to the original StockPriceService. The input variable is named StockSymbol, and the output is returned in StockPrice. The only important difference is that StockPriceService2 uses a different URL for stock information. All that remains is to add a single tag to the original StockPriceService output binding: <CONDITION TYPE="Failure" SERVICE="StockPriceService2"/>, which causes StockPriceService to chain to StockPriceService2 in the event that I am unable to retrieve a stock price from http://quote.yahoo.com/.
Making the Connection
Middleware insulates consumers and service suppliers from each other using a three-tier architecture -- the consumer of a WIDL service doesn't parse or "invoke" the WIDL file directly, but instructs a middle tier component to process the WIDL file. The middle tier component reads in the file, parses the XML tags, invokes the appropriate HTTP services, parses the HTML according to WIDL output bindings, and returns the appropriate results. Consumers of the WIDL service don't see any of this; from their perspective, the whole thing is simply an API call to the middleware components.
In today's world, the WIDL middle tier is vendor specific. Just as you need a browser and editor to manipulate HTML, you need tools to manipulate WIDL. And just as the browser is obtained from a particular vendor, so too are WIDL tools. For this article, I use the Web Automation Toolkit from WebMethods, the creators of WIDL. While a particular vendor's tools are used to construct and interpret WIDL, WIDL itself is an open standard proposal and is, in principle, platform- and vendor-neutral.
The Web Automation Toolkit 2.0 generates C/C++, Java, JavaScript, and Visual Basic code that encapsulates web resources, providing programmatic access to web data and services -- with or without a browser. The Toolkit automates access from applications to all web addresses (URLs), delivering HTML/XML parsing along with an object repository and the WIDL that maps document objects into program variables. The Toolkit includes wizards for creating WIDL files, and a user interface which graphically represents objects that may be programmatically accessed from static and dynamic web documents.
Using Web Automation, I've used the quote.widl file to generate a Java stub. The stub is used to invoke the WIDL processing. The code for this stub is shown in the class named QuoteStockPriceService.java (available electronically). For convenience, I've also included the files QuoteStockPriceServiceApplet.java and QuoteStockPriceService.html, which can be used to invoke the StockPriceService.
After all that preparation, you can now consume our service. As you can see from Listing Five, using the service is trivial. You simply instantiate a class and retrieve the stock value. No HTTP or HTML is visible. You don't have to recompile our client code if the stock service moves or changes. You have central administration control. We've turned an oblivious web page into a programmatic service that you can consume.
Future Directions
A number of additional XML-based data languages are being developed parallel to WIDL. CDF from Microsoft and the IETF WebDAV working drafts are example XML-based content languages being paired with HTTP protocols. The adoption of some of these proposals will expand the reach of the Web in ways we can scarcely imagine. Today's Web is highly interactive, but it is also largely a read-only, human-driven system. Tomorrow's Web will certainly be read-write and maybe navigable from software if systems like WIDL are widely adopted.
WIDL is still only a proposal; it might not be adopted by the W3C and it may or may not undergo significant changes. Nevertheless, the need for higher order consumption of web resources is becoming apparent. Over time, that need will be filled. The rapid adoption of component-based architectures multiplies the potential of automated access to the Web and facilitates the adoption of the same.
DDJ
Listing One
<?XML VERSION="1.0"?>< !DOCTYPE WIDL SYSTEM "widl.dtd"> < WIDL NAME="Quote" VERSION="2.0"> < !-- ---------------------------------------------- -->< SERVICE NAME="StockPriceService" METHOD="GET" URL="http://quote.yahoo.com/q" INPUT="StockPriceServiceInput" OUTPUT="StockPriceServiceOutput" />
Listing Two
<BINDING NAME="StockPriceServiceInput" TYPE="INPUT">
<VARIABLE NAME="StockSymbol" TYPE="String" FORMNAME="s" USAGE="DEFAULT" />
<VARIABLE NAME="DesiredDetail" TYPE="String" FORMNAME="d"
VALUE="v1" USAGE="DEFAULT" COMMENT="Basic " /><
/BINDING>
Listing Three
<BINDING NAME="StockPriceServiceOutput" TYPE="OUTPUT">
<VARIABLE NAME="StockPrice" TYPE="String" REFERENCE=
"doc.html[0].body[0].p[3].table[0].tr[1].td[2].text" USAGE="DEFAULT" /><
/BINDING>
Listing Four
<SERVICE NAME="StockPriceService2" METHOD="GET" URL="http://fast.quote.com/fq/quotecom/quote" INPUT="StockPriceService2Input" OUTPUT="StockPriceService2Output" /> < !-- ---------------------------------------------- --> < BINDING NAME="StockPriceService2Input" TYPE="INPUT"> <VARIABLE NAME="StockSymbol" TYPE="String" FORMNAME="symbols" USAGE="DEFAULT" />< /BINDING> < BINDING NAME="StockPriceService2Output" TYPE="OUTPUT"> <VARIABLE NAME="StockPrice" TYPE="String" REFERENCE="doc.table['*Last Sale*'].tr[2].td[1].text" USAGE="DEFAULT" />< /BINDING>
Listing Five
// This is a code fragment showing the use of the StockPriceService.
try
{
QuoteStockPriceService o = new QuoteStockPriceService(
StockSymbol
); System.out.println("StockPrice = " + o.StockPrice);
} catch (Exception e) {
System.err.println(e);
}
Copyright © 1998, Dr. Dobb's Journal
