Michael is DDJ's editor-at-large and can be contacted at [email protected].
The following is a case study of how one weekend web warrior tried to replace a rat's nest of utility web-building scripts written in an antiquated language, with modern technology, and the limited success with which his efforts were met. Along the way, it offers moral support for unrepentant HyperTalk scripters; surveys the relative merits of Python, Perl, and Tcl/Tk for web scripting; reflects on the difficulty of evicting poorly documented legacy code when carrying out software slum clearance in preparation for a high-rise software construction project; and gives an introduction to the e-mail handling capabilities of the Rebol scripting language.
The Situation
In a series of weak moments that should tell you all you need to know about my judgment, I agreed to take on the responsibility for creating and maintaining three web sites and updating them daily. Since my real work is writing prose, not HTML, I try to make this web site maintaining and updating as simple and automatic as possible, and I've created an assortment of tools toward that end.
I use an ISP, so updating my site means FTPing files over a standard (and slow) dial-up connection. As for computers, most of the machines I maintain in my and my partner Nancy's offices are Macs, so I use HyperCard and its scripting language HyperTalk for most of the utility scripts that I write. I make no apology for this, but an explanation is appropriate.
Although HyperCard has some limitations that have become frustrating and embarrassing over the years, third-party add-ons and stacks make it possible to do a lot with it. A HyperTalk script can launch other applications, handing off a URL to a browser for example, and individual HyperCard objects can be scripted in other OSA-compatible languages -- so while one HyperCard button might fire up a HyperTalk script when clicked, the button next to it might use AppleScript to employ capabilities that HyperTalk lacks. And I like it that there are so many stacks to steal good ideas from. An open-software culture has grown up around HyperCard, often encouraged or never actively discouraged by Apple. Also, for simple tasks, much of the coding is done before you start -- a HyperCard stack is already a flat-file database, for example.
But most importantly, I wrote one of the first books on HyperTalk and know the language inside out. When you spend a year of your life writing Dr. Dobb's Essential Hammer Handbook, everything comes to look like a nail.
So when I want to create some new tool to help manage my web sites, I write another HyperCard stack. (An aside to those for whom "stack" only signifies a LIFO queue: In the HyperCard world, the unit of application-level functionality is called a "stack." To the operating system, it's a HyperCard document; to the stack developer, it's a program. It's called a stack because it automatically employs the metaphor of a stack of index cards.)
Individually, these tools serve my needs just fine. However, they don't always play well with one another. HyperTalk is an old language. It has a limited form of inheritance, but it's not really object-oriented, and any connection between one stack and another has to be consciously programmed and maintained. When I ran into yet another problem caused by this limitation, I thought that it might be an appropriate time to look at alternatives to HyperTalk, and I began searching for a new scripting language.
The Tasks
If this is a case study, let me get down to cases and describe my web sites and the tools (stacks) I use to maintain them.
My web site is called Swaine's World (http://www.swaine.com/). There are three significant features of the site: a reader forum, an events calendar, and a news digest.
The news digest consists of a front page and several platform-specific or subject-specific news archives, each updated every weekday unless I get too busy. The front page includes headlines, quotes, and links for 15-30 news stories that have appeared online in the past 48 hours or so. The archives track a particular topic (UNIX/Linux news, international news, MP3 news, e-commerce news, and so on) over a longer period. Recently, I had also been beta-testing an e-mail version of the front page, which necessitates maintaining a list of e-mail addresses, reformatting the news digest in an e-mail-friendly form, and doing a daily mass e-mailing to the list.
The calendar is a graphical calendar display of shows and conferences for the coming two months. It's both a set of pointers to events and a tool for collecting events: Click on the event and you go to the event site, click on the date and you can send me information about an event for that date. To maintain this page, I have to put up a new calendar layout every month, post new events when I hear about them, and do some active searching for technology events.
The forum is far less well defined, and I have, in fact, yanked it offline until I can figure out how it ought to work and get it implemented properly. The idea is simple enough: It is supposed to be a place for readers of my columns to offer public feedback and perhaps start discussions, and a place for me to provide extra information in support of my columns. There is no lack of examples of such forums to examine, but I found something to dislike about every example I saw. Because I would be doing all the work of maintaining the forum myself, I had some pretty specific concerns regarding the efficiency of the processes involved. What, for example, is the most efficient way to ensure that I have a reader's permission to use his or her words, and that the reader understands that he or she has granted that permission, including permission to edit heavily? There are easy answers to those questions and to most of the questions I had, but neither the design I created nor the tools I wrote were providing the answers.
My second web site (http://www .fireinthevalley.com/) is the web site for my book Fire in the Valley, and at the time I write this, it's just getting started, so we'll skip it.
The third site (http://www.summerjo .com/) is Summer Jo's, the web site of my partner Nancy's business.
Summer Jo's is a many-headed beast: an organic farm, demonstration herb garden, and restaurant. Nancy has plans to expand into other areas in the next few months, too, holding workshops and selling herbal remedies and other farm- and garden-related products both on-premises and online.
The Summer Jo's site has dozens of pages, including a calendar that tells about upcoming events at Summer Jo's.
Another page that requires regular updating is the restaurant's Daily Specials page. The online catalog is a project unto itself (a whole web site and a whole business unto itself, really), and I've only begun work on that.
The Tools
Like every webmaster I know, I began creating and maintaining these pages using a text editor. Eventually, I began adding tools to automate parts of the process. One annoyance was having to change the navigation panel on every page whenever I added a new page to a site. Finally, I deconstructed the pages into a template and pieces, created an <include> tag to indicate where to put the pieces, and wrote my Site Maker stack to assemble the pieces of all the pages, according to the appropriate template, into the final HTML.
Building an HTML table from scratch for the calendar each month wasn't appealing, so I wrote a Calendar Maker stack to do that. Given a month, a year, and a list of events, it generates an HTML calendar component that can be dropped into most any web page, so it works for my two-month Swaine's World Tech Events calendar as well as the three-month Summer Jo's Farm Calendar.
And when Nancy started e-mailing me the daily specials for the restaurant to post on the site, I wrote a Specials Maker stack to grab that e-mail, convert it to HTML, and paste it into the right page at the right place. And so I went on, adding a new tool whenever a new problem came up, with little concern for how the tools worked together. Embarrassing, but true.
The most complicated stack is my News Maker stack, which I use to gather and format news items. It lets me click on a source, say the New York Times, and it sends the URL to my browser to take me to today's Times online. Once there, I spot an interesting news story, grab the URL, copy or write my own headline, grab a fair-use quote, and paste these data into fields in News Grabber, which automatically does some basic editing on the fields, removing excess spaces, fixing the capitalization and punctuation if I've grabbed less than a complete sentence, checking for a well-formed URL, and so on. News Grabber maintains a database of hundreds of news sources with an average of a dozen or so writers for each, so I can usually just pick the source and writer from a list. New writers or sources are added to the database automatically. I pick one or more keywords from another list, and if there's a good graphic, I paste its URL into another field.
When I'm done grabbing the stories, I write a brief summary, reorder the stories by clicking on their headlines in a field, and click a few more buttons to produce the HTML for the front page, the archive pages, and the newsletter version. Up until a month ago, that was the end of the process.
When I deconstructed the Summer Jo's pages into template and pieces, it made changing that site easy, but my Swaine's World pages were still undeconstructed. I wanted to use Site Maker with them so that I could change the navigation panel or logo without having to edit every page by hand, but it wasn't at all clear that if I deconstructed them, the resulting pages generated by my Site Maker stack would still work with my News Maker stack. It turned out there was no incompatibility and the tools worked together just fine, but I considered that more a matter of luck than good programming.
And I regarded this experience as a warning. The sites were getting bigger, and with the upcoming online catalog, I was getting to the point where I couldn't afford a day of downtime. I was feeling the need for some better organization of these tools.
The Search
I thought I'd try replacing my web-building tools with a new scripting language one tool at a time. Maybe I'd stay with HyperCard for some of them, maybe not. The search immersed me in the worlds of Perl, Python, and Tcl/Tk. Conveniently for me, conferences on all three languages were held at the same time in Monterey this past summer, and I added some workshop sessions to my researches into the three scripting languages.
The virtues of Perl are well known. It is, according to its author, the first postmodern programming language, and I'm sure that means something important. It's also the most popular scripting language today and, also according to its author, the language of choice for network abuse. Spam, that is. Since my e-mailer is the moral equivalent of spam (an opt-in mass e-mailer), why not use Perl? I couldn't reject it immediately, so I looked at the others.
I considered the virtues of Tcl/Tk, including its extensibility, the ability to use it as an integrator of applications. If my HyperTalk routines could be turned into Tcl extensions, I could use Tcl for tying together the components. But they couldn't, that was not exactly the problem I was trying to solve, and I couldn't see that Tcl itself was the tool for writing the new components.
Then there was Python: portable, which my HyperCard stacks are not, extensible to a degree that HyperCard is not, and with support for object-oriented development. I had to take it seriously. But if I was going to commit to one new scripting language, I suspected that Perl was going to win over Python, if only because it's so much more established.
I already code in JavaScript and considered it briefly for the projects, but it wasn't really appropriate for the purposes I had in mind.
But in the end, I didn't use any of these languages. At the same time that I was evaluating them, I was working with the Rebol scripting language for some articles I was writing. In looking over the projects, I got to the point where I could envision how to implement them in Rebol. Once I could see the path, I didn't look back. I dropped the alternatives and went to work on the Rebol implementation of several components. I don't pretend that this was a deliberate, carefully considered decision. But this is a case study and this is what I did.
It wasn't a bad decision. For certain kinds of tasks, Rebol is easier to use than any of the aforementioned other tools. And the kinds of tasks that Rebol makes particularly easy are tasks involving Internet programming -- e-mail, HTML, and FTP processing -- just the kinds of tasks I had in mind. Want to grab a page off the Web and e-mail its contents to several friends? That takes just a couple of lines in Rebol:
list make block! [[email protected] nancy@ summerjo.com]
send list read http://www.fireinthevalley .com/index.html
I don't think any of the other scripting languages make this sort of thing this easy. Then, too, I'm a lifetime student and Rebol promised to teach me some new tricks. Maybe you'll find some new tricks in it, too.
The Rebol Scripts
Initially, I decided to use Rebol for the Forum and newsletter. I had high hopes based on its strengths in managing Internet protocols.
Rebol understands a lot about a lot of Internet protocols. Using the pop protocol, for example, you can examine your e-mail on the server before downloading it. It is also useful for creating custom e-mail filters.
As I dug into it, I began to realize that one reason I hadn't liked any forum I'd seen was that what I really wanted to do couldn't be done in one forum: I needed at least a couple different kinds of user forums to implement the different kinds of interaction I had in mind.
One forum I had in mind would allow visitors to comment on my writing. Rebol's creator, Carl Sassenrath, sent me a script that addressed that need. I've tweaked it slightly and it's what you see in Listing One. Along with the web page HTML code in Listing Two, it sets up a simple arrangement that allows visitors to a page to add comments to an article on that page. You can see the page as Carl designed it in Figure 1.
One interesting line in Listing One is the one that starts out cgi: make object! decode-cgi...
Rebol knows all about cgi query strings and knows how to decode them. Other programs at the Rebol site (http://www.rebol .com/) show off its CGI-handling capabilities even more.
Listing Three is part of the newsletter solution. All it does is to send various versions of a file to a database of e-mail addresses. Very few of the lines in this listing are needed for actually sending the different newsletters to the different groups of subscribers.
As of this writing, I am still evaluating the Rebol tools, still looking for the weak elements in my collection of site-maintenance tools, still worrying about how the pieces work together, and still considering using Perl for part of the solution. I've learned some things about these scripting languages, but probably haven't learned much new about programming methodology. Maybe that's because so far all these tools are working. If everything is still working when this article goes to press, you can see what progress I've made at the web sites mentioned.
DDJ
Listing One
#!rebol -cs
;-- The above lines are used by Apache CGI to launch REBOL and run this
; script. The -cs is used to indicate that it is a CGI script and that
; security is lowered to allow REBOL to read and write local files.
REBOL [
Title: "CGI Web Page Comment Poster"
Date: 1-September-1999
Needs: 2.1.2
Note: {
For each article, in the comment posting form, be sure
to modify the hidden "file" field value to give it the
correct name of the HTML file to receive the comments.
Also, the HTML file must have an HTML comment to note
where the table insertion is made. See comments below.
}
]
;-- A function to create the HTML used for the comments.
make-comment: func [from date comment] [
;-- This function, which just returns some HTML, has been deleted.
]
;-- Process the CGI query string, making an object from its fields.
cgi: make object! decode-cgi system/options/cgi/query-string
;-- The file name of the article is provided in a hidden input field within
; field within the HTML of the article. Use this string to build the path
; to the file from CGI dir. Remember file must have write permissions
; if you want add comments.
file: join %../web/ cgi/file
;-- Create the text of the new comment from the CGI input.
; If the type is code, then display it as preformated TTY.
new-comment: make-comment cgi/from now either cgi/type = "code" [
rejoin ["<pre>" cgi/comment </pre>]][cgi/comment]
;-- Read the HTML file, add the newest comment to it, and write
; it out. An HTML comment is used to mark where it goes.
page: read file
insert find page <!--comments--> new-comment
write file page
;-- Display the page again:
print page
Listing Two
<html>
<head>
<meta http-equiv="Content-type" content="text/html;charset=iso-8859-1">
<meta name="Author" Content="carl">
<title>Article</title>
</head>
<body bgcolor="white">
<form action="/cgi-bin/comment.r" method="get">
<h3><font face="Arial, Helvetica">Article</font></h3>
<blockquote>
<p>Text of the article would go here. This entire page can use
whatever type of formatting you prefer. However, remember that the comments
table below must have an HTML comment to indicate where new messages are
placed, and the "post" form must have a hidden input field to
relay the article's file name. <i>Do not forget to change this field or
comments will be posted to the wrong page</i>.</p>
<p><br>
<table border="0" cellpadding="2" cellspacing="1" width="80%">
<tr>
<td colspan="3" bgcolor="navy">
<P align="center"><b><font size="2"
color="white" face="Arial, Helvetica">COMMENTS</font></b>
</td>
</tr>
<!--comments-->
</table>
</p>
<p>
<table border="0" cellpadding="2" cellspacing="1" width="80%">
<tr>
<td width="10%" valign="middle" bgcolor="#660000">
<P align="right"><b><font size="2"
color="white" face="Arial, Helvetica">From:</font></b>
</td>
<td width="86%" valign="middle"
bgcolor="#EBC2A7"><input type="text" name="from" size="50"></td>
</tr>
<tr>
<td width="10%" bgcolor="#660000">
<P align="right"><b><font size="2"
color="white" face="Arial, Helvetica">Comment:</font></b>
</td>
<td width="86%" valign="middle"
bgcolor="#EBC2A7"><textarea name="comment"rows="10"cols="50"></textarea></td>
</tr>
<tr>
<td width="10%" bgcolor="#660000">
<P align="right"><b><font size="2"
color="white" face="Arial, Helvetica">type:</font></b>
</td>
<td width="86%" bgcolor="#EBC2A7">
<center>
<p><input type="radio" name="type"
value="regular" checked><font size="2" face="Arial, Helvetica">Regular Text
<input type="radio" name="type"
value="code">Code Listing (preformatted)</font>
</center>
</td>
</tr>
<tr>
<td width="10%"></td>
<td width="86%">
<center>
<p><input type="submit" name="topic"
value="Post Comment">
</center>
</td>
</tr>
</table>
<input type="hidden" name="file" size="-1" value="article.html">
</blockquote>
</form>
</body>
</html>
Listing Three
REBOL [
Title: "Custom Newsletter Emailer"
Date: "September 1, 1999"
File: %customnewsmailer.r
Purpose: "Emails different versions of newsletter to specified subscribers."
]
{Subscribers is database of subscribers. We read it from a cdf file and
convert to a Rebol block. Format of each record, which identifies one
subscriber, is: subscriber-group, first-name, last-name, email.
Subscriber-group is a number specifying which newsletter version subscriber
gets; the number corresponds to the newsletter version's filename extension.}
subscriber-database: read/lines %subscribers.txt
subscribers: make block! []
foreach line next subscriber-database [append subscribers parse/all line ","
]
{Newsletters is number of different versions of newsletter we have.}
newsletters: 2
{step through the newsletter versions}
repeat next-letter newsletters [
{read in the next version of the newsletter from a file}
newsletter: read/string to-file join "newsletter." next-letter
{step through the subscribers}
foreach [subscriber-group first-name last-name email] subscribers [
if (to-integer subscriber-group) = next-letter [
bind subscribers 'email ; any word in local context will do
print ["Sending newsletter version" next-letter "to:"
first-name last-name]
{build mail header needed to send newsletter to self, bcc to list}
header: make system/standard/email [
from: (system/user/email)
to: (system/user/email)
bcc: (reduce [email])
subject: "Swaine's World Technewsletter"
]
{send this version of the newsletter to all the appropriate folks}
send/header/only join header/bcc header/to newsletter header
]
]
]
