Building and Running Online Auctions

Webalog, a tool for enhancing online commerce

Brent is president of Bonsai Software and codesigner of Webalog. He can be contacted at [email protected]. Greg works for Visible Decisions and can be contacted at [email protected].

With the exceptions of sites such as Amazon.com (http://www.amazon.com/) and Firefly (http:// www.firefly.com/), web-based retailing hasn't really taken off. One reason is that electronic cash is still in its infancy, and many people are reluctant to send credit card numbers over the Internet. A more fundamental reason is that web-based shopping isn't actually much of an improvement over more traditional means. Browsing a catalog online is usually slower than flipping through one made of dead trees, and ordering something via the Internet is rarely quicker than calling a toll-free number. The only real advantage of online catalogs is that they are always up-to-date, but this is not a major concern in most traditional retailing, which tends to be highly seasonal. Few people need to know, for example, exactly when a retailer runs out of winter coats.

However, there is one area where the Web is clearly better than traditional alternatives -- auctions. An auction is inherently interactive, and involves many players instead of a conventional sale's single buyer and seller. Auctions also tend to deal in one-of-a-kind or close-out items, which has traditionally made it ineffective to produce catalogs for all but the highest-priced sales.

In this article, we'll describe a web-site toolkit called Webalog, developed by Bonsai Software, which is being used to construct online auctions and various similar web-based applications.

In an auction, users can browse the items for sale and enter a maximum (proxy) bid price. The Webalog server then plays out the bidding until at least one bidder's maximum price is reached. Other bidders are then asked whether they want to go higher. This cycle repeats until a time deadline is reached, or until an undisputed winner emerges. Online visitors can participate in multiple auctions at the same time, relying on Webalog to keep them abreast of their position in each.

Webalog not only demonstrates how commerce can take advantage of the Web's strengths, but also shows how freeware can be leveraged to build useful, flexible applications as new opportunities arise. In addition to being used for online auctions and catalogs, Webalog is the basis for a benefits-selection system, a biological data registry site, a time-sheet reporting system, a task-management system, and an intranet customer-list system. All of these applications were developed by just four developers in just two years. It is difficult to imagine how such a small team could construct so many diverse products so quickly if freeware had not made it as easy to "recycle" as to "build" or "buy."

How Webalog Works

Webalog works by embedding commands and data in HTML pages, then interpreting them on the fly. Commands are written in Tcl. Webalog's designers chose to use Tcl for three reasons:

• Tcl is in the public domain, and therefore available for use on almost any platform.
• Unlike languages such as Perl and Java, Tcl scripts are not compiled to bytecode before being run, but are instead interpreted directly. This speeds up the development/debug cycle and (more importantly) minimizes the startup delay of small applications.
• Tcl is extensible: You can add commands to the base language without any performance penalty by compiling in your own C libraries. It is also a glue language: It can easily be used to bind databases, existing libraries of code, and custom programs together.

Webalog was originally developed for building catalogs, but its authors soon found that there was considerably more interest in online auctions. One such auction, from which the examples given here are drawn, is at http://www.auctionx.com/ (see Figure 1). When users first retrieve the start page from this site, Webalog fetches a list of the categories of items currently up for auction from a SQL database and turns into HTML. As it does this, Webalog inserts a "session ID" into all of the URLs of that page. A session ID is simply a unique, machine-generated identifier; a new session ID is generated for each user on the first page fetch he or she makes each day, and is threaded into the URLs that Webalog cares about. Subsequent fetches use the session ID to keep track of the state of the session. For example, a typical URL within a page of the site will look like normal HTML:

<a href=/about.html>About</a> this site. 

After a session ID is inserted, this URL would appear as:

<a href=http://www.auctionx.com/q/@035790ztzzgv/about.html>About</a> this site.

Webalog's use of session IDs to track user operations highlights an aspect of the Web that is so obvious that it is often forgotten. Since the only part of the URL that is interpreted at the client side is the address of the machine to which the URL refers, site builders are free to use URLs as more than just pointers. URLs are not physical links to sites, any more than file names are directly mapped to sectors on a disk. Every time a page is fetched, several programs must cooperate to map the URL into a file, then pass the contents of that file back to the requesting browser. The power and flexibility of web-based systems can be greatly enhanced by modifying either the identifier (the URL) or any of the intervening programs.

Session IDs are used to keep track of subsequent page fetches from the same browser, threading the activity together into a session. With this information, Webalog can preload relevant state information from its session database. This database holds such things as the user's login name, his or her shopping basket (in the case of a catalog), and any other information the site builder decided to store. Any information stored to the session database is made available within the context of any other page for the duration of the session.

Webalog uses its own session IDs instead of Netscape cookies to keep track of users for two reasons. First, not all browsers support Netscape cookies, and Webalog was designed to run with everything, everywhere. While it would be possible to hide the Webalog session ID inside a Netscape cookie, this hasn't yet been important enough to merit development effort. The backlash against Netscape cookies is a second reason for using modified URLs instead: To site managers, being nonintrusive and out in the open is actually a selling point for Webalog.

Webalog's heart is a universal CGI-bin script called "QB" that runs on the server side of page fetch. As Figure 2 illustrates, QB sits between the server and the web site, watching the conversation, and helping out when asked.

When the server receives a page request, the URL is handed to QB first. QB parses the session ID contained in the URL, then loads the proper context for that session (such as the currently active shopping cart). Any variables from the command line are parsed out and bound to the environment as Tcl variables. QB then loads the requested HTML or CGI-bin file. As the information is passed back to the server for return to the user, QB interprets any Tcl scripts embedded in the page. This provides Webalog with its programmability and will be discussed in more detail shortly. Finally, QB parses the output stream, modifying any URLs it contains in order to thread the session as previously described.

As simple as it is, QB provides all the infrastructure required to build high-end sites, such as session tracking and programmability. As a bonus, this system automatically produces a session log file enabling detailed (and customizable) site-management reports. This information has been used to provide statistics on the typical path visitors take through a site and the amount of time spent in an area (such as the search or help section of the site). This allows site managers to spot unnecessarily long URL chains, or provide quick jumps to frequently visited pages. As the novelty of the Web wears off and competition between sites heats up, this kind of tuning can make the difference between commercial success and failure.

Embedding Tcl

What makes QB especially powerful as a site-development tool is that it allows any page to contain sections of Tcl that are executed in place during the page's return from the server. This gives developers a simple, well-known programming environment with which to do powerful scripting within the page. For example, if Listing One were embedded in a page, users reading that page would see the digits 0 through 9.

Like many systems, Webalog started by using programs to print out HTML. Though it was flexible and programmable, this method led to long chains of hard-to-read put statements (the Tcl equivalent of print). Webalog's designers eventually realized that turning this inside out -- that is, putting the Tcl in the HTML -- was more readable, hence easier for site managers to modify and maintain. It also allowed them to use WYSIWYG page-layout tools, which is a definite plus as more complicated effects are more widely used. Since the processing is done on the server side, the qbtcl tags are consumed, and browsers need not deal with them.

QB also has a simple mechanism for presenting live data using qbsub regions. QB replaces the text of any Tcl variables in a qbsub region with the values of those variables. While it is not as powerful as full-blown scripting, variable substitution enables nonprogrammers (such as graphic artists) to build pages as if they were pure HTML, while taking advantage of many of the capabilities provided by QB. If a bid ID and a manufacturer's description were saved in a session database, for example, Listing Two would be filled in with the appropriate data in the obvious way.

Variable substitution of this kind greatly enhances the ability of nontechnical webmasters to edit pages for look and feel, while leaving programmers free to work on the code for the logical data movement through the site. Values for variables can be looked up in a variety of databases; the list of currently supported databases includes db, bdb, dbm, Msql, and Informix.

QB uses the session ID for more than tracking; it also provides state for the session. The session ID points into a session database (sdb), using either of UNIX's db or bdb formats, containing information stored by the site programmer. Thus, a login page could include Listing Three, so that a subsequent page could use this information as part of a database lookup; in the case of Listing Four for fetching the user information.

Some Webalog users have asked why it uses Tcl, rather than Java or JavaScript. The simplest answer is that neither Java nor JavaScript is available on all interesting platforms (yet). In addition, for many applications, the processing needs to be done on the server side. Bid checking, for example, needs access to the current state of the auction and has to be able to lock the relevant database records for update. As exciting as Java might be, its "let the client compute" model is simply not appropriate for this problem.

Finally, there are some serious drawbacks to client-side Java. It could be used for simple tasks, such as validating form fields, but the caching features of some browsers can cause major problems. Once a user downloads an applet, the browser's cache wants to use that copy for a long time, which makes code updates problematic (unless something like Marimba's Castanet is used, but this, in turn, forces users to download yet another software set). Provider caches can be problematic as well. At one point, America Online changed its caching policy, negatively affecting many content providers. The fix was to have QB insert a simple cache-buster tag into each outgoing page. Other solutions may have required every page on the system to be edited to contain anticaching directives.

Forms and URL Arguments

Forms are the most widely used input mechanism on the Web; in fact, some would argue that forms and embedded graphics are what made the Web more than just a point-and-click interface to FTP. In a QB-based site, values in form submissions are automatically bound to the associative array named qb_val. A form submission can therefore be processed using something like Listing Five.

QB recognizes standard argument-passing tricks, such as embedding options in the URL. When such options are used, QB recognizes and binds the appropriate variable for the programmer. Consider the URL http://www.auctionx .com/bidform.html?lot_ id@663+login@tycho. Within the page "bidform.html," the Tcl lot_id variable would be created and bound to the value "663." Likewise, the Tcl login variable would be created with the value "tycho."

Finally, QB has a number of built-in commands, which take the form qb <command> <args>. Our experience is that any nontrivial interactive web site needs to have the capabilities provided by these functions, just as any nontrivial programming languages needs to support modules, debugging, safe concurrency, and referential data structures. Table 1 lists the most commonly used of these commands.

Conclusion

Webalog is a good example of "opportunistic" development. Originally intended for use in online catalogs, Webalog was instantly useful in developing online auctions when the demand for them emerged. This rapid development simply would not have been possible without the large amount of high-quality freeware that is now available.

One possible future direction for Webalog is to embed the technology in an HTTP server, further reducing the startup time and overhead involved. Another is to support live auctions through the Web. This would enable Internet users to participate in traditional auctions alongside present participants. This is an area where Java's client-side processing could be of use in combination with Webalog's server-side capabilities.

In the longer term, Webalog auctions are an example of what the agent-based future of networked computing may look like. Every time a user registers a bid, he or she is effectively creating a simple-minded agent that knows how much to offer for something, how aggressively to try to acquire it, and how much its principal is willing to pay. While the agents of the future might be free-floating bits of executable code, their behavior and utility will be similar to that of Webalog's bids.

For More Information

Bonsai Software
2582 Old First Street
Livermore, CA 94550-3155
510-606-5420
http://www.bonsai.com/

Listing One

<html><qbtcl>
    for {set i 0} { $i < 10 } { incr i } {
        puts "i is $i <br>"
    }
</qbtcl>
 ...
</html>

Back to Article

Listing Two

<qbsub>    Lot Number: $lotNum<br>
    Description (Manufacturer): $lotDesc <br>
    Quantity Available: $lotQuant <br>
    Current Price: $lotMinBid <br>
    ...
</qbsub>

Back to Article

Listing Three

<qbtcl>   set sdb(login) $username
</qbtcl>

Back to Article

Listing Four

<qbtcl>    if {info exists sdb(login)} {
       set userquery "select * from users where login = \"$sdb(login)\""
       set userhandle [ esql query $userquery ]
        ...
    }
</qbtcl>

Back to Article

Listing Five

  ...<qbtcl>
    # was a login provided?
    if [ info exists qb_val(login) ] {
        set sdb(login) $qb_val(login)  # remember this for future use
    }
</qbtcl>

Back to Article

DDJ