HTML Templates for Lisp

An elegant language leads to an elegant solution

Gene is a software engineering consultant in Seattle. He can be contacted at geneacm.org.

While a program might generate HTML with a bunch of print statements, most programmers who have tried such techniques agree that the code becomes ugly and unmanageable if the HTML page is at all complex. Luckily, HTML templates provide an alternate way of generating HTML. Instead of writing code that produces HTML, you write a template file that contains verbatim HTML but can escape out of HTML mode and into a programming language mode to generate some parts of the HTML document. PHP is often used for HTML templates. There are HTML template libraries for Perl and other languages. However, I wanted an HTML template library for Lisp because I write CGI programs in Lisp and because HTML templates can integrate into Lisp almost transparently. The complete library is available electronically; see "Resource Center," page 5.

Listing One is an example of a template. It starts with plain HTML, but there's Lisp code between <lisp> and </lisp> tags. If a chunk of Lisp code returns a string, the HTML template library writes that string to the same place it writes the literal HTML text. To use this HTML template on a web site, a Lisp CGI program would send the HTML parts verbatim, but it would process the Lisp code within the <lisp> and </lisp> tags and send their results to the web browser. The web browser would never see the <lisp> and </lisp> tags, nor the Lisp code between them. The <lisp>...</lisp> sections cannot overlap or nest, and all <lisp> tags must be closed by </lisp> tags.

Top-Level Usage

Listing Two illustrates how a program might use an HTML template library. The first part of Listing Two shows how a program could call a function named CALL-TEMPLATE to execute an HTML template. The Lisp code within the template could fetch parameters from global variables or from the GET or POST arguments in an HTTP request.

The next part of Listing Two shows use of the INPLACE-TEMPLATE macro. It would convert the HTML template into Lisp code that is inserted where the program calls INPLACE-TEMPLATE, so the Lisp code from the template could refer to local variables where INPLACE-TEMPLATE was used. This was the feature I originally wanted in an HTML template library for Lisp.

The final part of Listing Two shows use of a DEF-TEMPLATE-FUN macro to convert the HTML template into a Lisp function that could be called by name. After using this HTML template library, this became my favorite way of using templates.

Helper Functions

I'll implement this HTML template library from the bottom and work up to the top-level functions. Because space is limited and these functions are straightforward, I don't show the complete source code here but, again, it is available electronically.

You need a function to read the entire contents of a file and return it as a string. I call that function SLURP-FILE. It has one argument, which is the pathname of a file to read. SLURP-FILE opens the file, reads its contents into a string, and returns the string.

For parsing the HTML template file, you need a SPLITS function. Its first argument is a substring and its second argument is a big string. SPLITS returns a list of the parts of the big string that were separated by the substring.

Another function you need is SEND, which has two arguments—a string (or sequence of strings) and a stream on which to write the string (or strings). SEND recursively decomposes and examines its first argument. Whenever it finds a string or other atom, it prints those. So each verbatim HTML section of a template file becomes an argument to SEND, and the Lisp parts of the template file can return a string or a list of strings for SEND to print. (The Lisp parts could write their own output and then return an empty list or an empty string so that SEND will print nothing.)

Once you read an HTML template file's contents into a string, you need to preserve the verbatim HTML parts as strings and convert the Lisp code chunks into lists of Lisp expressions. One way to separate the HTML template into HTML parts and Lisp parts is to split it on the </lisp> tags, then split the results on their <lisp> tags. Then you have a list of two-element lists. The first item of each sublist is verbatim HTML. The second item of each sublist is Lisp code in a string. The function that does all this is SPLIT-TEMPLATE-INTO-PAIRS.

Function READ-TEMPLATE

Given the contents of an HTML template file as a string, READ-TEMPLATE returns a list of Lisp expressions derived from the HTML template string; see Listing Three. It is the output of calling READ-TEMPLATE on the HTML template from Listing One. Each verbatim HTML part from the template string becomes the argument to SEND. Each chunk of Lisp code from the template string becomes multiple Lisp expressions, and their result becomes the argument to SEND. So the output of READ-TEMPLATE is a list of SEND function calls. Listing Four shows READ-TEMPLATE.

You might notice that READ-TEMPLATE forms SEND calls with two arguments. The first is a string or list of strings, as I've already explained. The second is STRM, which isn't necessary but lets the program specify a destination stream for SEND; otherwise, SEND writes the strings to standard output.

Also notice that I've talked about converting strings to Lisp, but so far, it's only converted to Lisp data. READ-TEMPLATE returns a list of Lisp data, not Lisp code. It is the responsibility of READ-TEMPLATE's caller to convert the data to code.

CALL-TEMPLATE and LOAD-TEMPLATE

One way for programs to use HTML templates is to name the template file in an argument to a function that executes the template. CALL-TEMPLATE is such a template-executing function, and it lets LOAD-TEMPLATE do some of the work. Listing Five shows both functions. LOAD-TEMPLATE uses READ-TEMPLATE to convert the HTML template file to a Lisp function, then returns it. To use an HTML template just once, it is easier to use CALL-TEMPLATE to load and run the function with one line of code, but if a program needed to use the HTML template many times, it would be more efficient to save the function from LOAD-TEMPLATE into a variable, then call the function from the variable.

Listing Six is an example of using CALL-TEMPLATE. The program generates two HTML files from a single HTML template using a different list of numbers for each file. The program uses the HTML template file in Listing Seven. The Lisp code in the template writes directly to the output stream, STRM, which is an argument of the unnamed function that LOAD-TEMPLATE created from the HTML template file. Alternatively, the Lisp code could have collected its output into a string and returned that, but doing so would have used WITH-OUTPUT-TO-STRING, which would have been one more line of code.

The EVAL in LOAD-TEMPLATE might raise warnings for experienced Lisp programmers. It's usually better to use a macro than EVAL, but that won't work for LOAD-TEMPLATE. Macros manipulate Lisp code as data. The arguments to a macro must be literal code, but in this case, we have Lisp data, which was determined at runtime. A macro won't work unless we make a sacrifice.

The INPLACE-TEMPLATE Macro

For a macro to work, the pathname must be a literal. If the pathname to the file is a literal, you can dispense with the EVAL function and use a macro to convert the template into Lisp code. Listing Eight is an INPLACE-TEMPLATE macro.

Instead of calling EVAL, INPLACE-TEMPLATE returns an anonymous function in an expression to call it. (It could have returned a LET instead.) Lisp evaluates that expression in-place, inside the lexical context where INPLACE-TEMPLATE was called, so the anonymous function can access local variables just as a nested block of code could. Listing Nine shows how to use INPLACE-TEMPLATE. Because the macro's result is within the current lexical environment, the Lisp code in the HTML template can refer to local variables, and there is no need for an optional HT argument to the INPLACE-TEMPLATE macro or to the function it returns.

The disadvantage to INPLACE-TEMPLATE is that the pathname of the HTML template file must be a literal such as "myfile.tmpl." If the pathname argument to INPLACE-TEMPLATE is an expression such as (make-pathname :name "myfile" :type "tmpl"), which returns a pathname, INPLACE-TEMPLATE will fail because the MAKE-PATHNAME expression won't be evaluated. A workaround might be to use a literal LOGICAL pathname. Logical pathnames are a system of portable pathnames in Lisp; they translate to local pathnames through a configuration that is created when the program is installed. If an HTML template file is considered a sort of source file, it would be a good idea to load it with a logical pathname.

HTML Templates as Named Functions

It's possible and sometimes most convenient to call a named Lisp function that does the work of a template instead of using CALL-TEMPLATE or INPLACE-TEMPLATE. One way to do that is to create named Lisp functions that wrap calls to INPLACE-TEMPLATE or CALL-TEMPLATE, but it would be easier to let Lisp do the work.

DEF-TEMPLATE-FUN (Listing Ten) is a macro that let's Lisp do that work. It's like INPLACE-TEMPLATE, but instead of returning an anonymous function, DEF-TEMPLATE-FUN returns an expression that creates a named function. DEF-TEMPLATE-FUN's first two arguments are like DEFUN's first two: function name and argument list. Instead of a function body for the rest of its arguments, DEF-TEMPLATE-FUN needs the pathname of the HTML template file.

Instead of using DEF-TEMPLATE-FUN directly, Lisp can do more of the work. DEF-ALL-TEMPLATES is a function that locates all the HTML template files that match a pathname containing wildcards, then creates and evaluates expressions that use DEF-TEMPLATE-FUN to create functions for each of the HTML template files. DEF-ALL-TEMPLATES derives the function name from the template filename, and it assumes that the function has two optional arguments: a stream and a second argument (which I usually make a hash table to carry multiple values).

Listing Eleven shows DEF-ALL-TEMPLATES. You might notice that it calls EVAL. My experiences brought me back to using a function so that I wouldn't be limited to literal pathnames. The disadvantage is that the functions it creates are defined at top level, so they can't share local variables. So far, that hasn't been a problem.

The function is simple. It uses DIRECTORY to get all the pathnames that match its argument, then it constructs a DEF-TEMPLATE-FUN expression around each of those pathnames. It wraps all the DEF-TEMPLATE-FUN expressions in a PROGN so they become a single expression, and it uses EVAL to execute them. All those DEF-TEMPLATE-FUN expressions create Lisp functions as already described.

Assuming there is a function GET-CGI-ARG, which returns the value for an argument sent by a web browser, and assuming there are a lot of HTML templates in a templates/ directory, Listing Twelve shows how a program might put DEF-ALL-TEMPLATES to use.

Conclusions

So that's an HTML template library for Lisp. I use it for some CGI programs and some offline programs that run when I compile my web site. All in all, the library is about 200 lines of code. Additionally, you can find CGI programs that use the library at http://lisp-p.org/lahte/.

DDJ

Listing One

<html>
<!--
  Lisp expects two parameters as global variables:
  *TITLE* is the title as a string.
  *N-LST* is a list of numbers.
-->
<head><title><lisp>*title*</lisp></title></head>
<body>
<table>
<tr><th>n</th><th>n<sup>2</sup></th></tr>
<lisp>
  ;; Generate HTML and collect it into a string,
  ;; then return the string.
  (with-output-to-string (strm)
    (dolist (n n-lst)
      (format strm "~&<tr><td>~A</td><td>~A</td></tr>"
              n (expt n 2))))
</lisp>
</table></body></html>

;; Example of using HTML templates
;; Dynamically transform the template into a function & call it.
(call-template "potato.tmpl")

;; Transform the template into a function in-place
(let ((*i* 0))
  (inplace-template "potato.tmpl")
  ;; The template could use & alter the value of I.
  (format t "*I* is ~A" *i*))

;; Load the template into a function at compile-time
(def-template-fun potato (arg0 arg1 arg2) "potato.tmpl")
;; then call it as a regular Lisp function
(potato 1 2 3)

;; Example of what READ-TEMPLATE might return
((send "<html>
<!--
  Lisp expects two parameters as global variables:
  *TITLE* is the title as a string.
  *N-LST* is a list of numbers.
-->
<head><title>")
 (send (progn *title*))
 (send "</title></head>
<body>
<table>
<tr><th>n</th><th>n<sup>2</sup></th></tr>
")
 (send (progn (with-output-to-string (strm)
    (dolist (n n-lst)
      (format strm "~&<tr><td>~A</td><td>~A</td></tr>"
              n (expt n 2))))))
 (send "
</table></body></html>
"))

(defun read-template (template)
  "Returns a Lisp datum derived from the HTML template, which is a string."
  (mappend #'(lambda (lst2)
           (list `(send ,(first lst2) strm)
             (read-from-string 
              (format nil "(send (progn~%~A~%) strm)"
                  (second lst2)))))
       (split-template-into-pairs template)))

(defun call-template (pn &optional (strm *standard-output*) ht)
  "Given the pathname of an HTML template file, convert the file to a Lisp
function & call it. STRM is the destination of the template's HTML output.
HT is an extra argument for the Lisp code in the template.  The idea is that
HT is a hash table."
  (funcall (load-template pn) strm ht))
(defun load-template (pn)
  "Convert the HTML template to a function & return the closure.  The
function is defined at top level."
  (eval `#'(lambda (&optional (strm *standard-output*) ht)
         ,(read-template (slurp-file pn)))))

;; Program to write two HTML files containing some tabular data.
(defun example (n-lst pn)
  (with-open-file (strm pn :direction :output)
    (call-template "listing-07.tmpl" strm n-lst)))
(example "units.html" '(1 2 3))
(example "tens.html" '(10 20 30))

<html><head><title>table of numbers</title></head><body><table>
<tr><th>N</th> <th>Log N</th> <th>N<sup>2</sup></th></tr>
<lisp>
  (dolist (n ht)
    (format strm "~&<tr>~{<td>~A</td>~}</tr>"
            n (log n) (expt n 2)))
</lisp>
</table>
</body></html>

(defmacro inplace-template (pn &optional (strm *standard-output*))
  `(funcall #'(lambda (strm) ,(read-template (slurp-file pn)))
        ,strm))

;; Use INPLACE-TEMPLATE so an HTML template function can
;; access local variables.  If the HTML template file holds this:
;;   <p>X is <lisp>x</lisp>.</p>
;; then
(let ((x 42))
  ;; this template function will show 42,
  (inplace-template "the-template.tmpl"))
;; but this template function will FAIL because X is not in scope.
(inplace-template "the-template.tmpl")

(defmacro def-template-fun (name args pn)
  "From the HTML template file PN create a function called NAME.  ARGS should
contain a STRM formal argument. It could contain other formal arguments, too.
(A better implementation of this function would insert the STRM argument if
it was missing.)"
  `(defun ,name ,args
     ,(format nil "Created from the HTML template ~S." pn)
     ,(read-template (slurp-file pn))
     ',name))

(defun def-all-templates (&optional (dir *def-all-templates-default*))
  (eval `(progn
       ,(mapcar #'(lambda (pn)
             `(def-template-fun
                ,(template-name-from-pathname pn)
                (&optional (strm *standard-output*) (ht nil))
                ,pn))
             (directory dir)))))

#! /usr/local/bin/clisp
;; An example CGI program
(format t "Content-type: text/html~%~%")

;; Load libraries & such, including the HTML template library.
(setq *load-verbose* nil)
(load "all-that-template-stuff.lisp")

;; Convert all the HTML template files into named Lisp functions.  Assume
;; the templates are called MYINSERT, MYDELETE, MYEDIT, and MYERROR.
(def-all-templates (make-pathname :directory '(:relative "templates")
                  :name :wild :type "tmpl"))
;; a very simple "main"
(let ((action (get-cgi-arg "action")))
  (cond ((equal action "insert") (myinsert))
        ((equal action "delete") (mydelete))
        ((equal action "edit") (myedit))
        (t (myerror))))