Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼

RESTful Web Service in Go Powered by the Google App Engine

In the next lines of code, I iterate over the all found strings and check whether the current one looks as a batch number. If yes, I take it along with the next two and put them aside as a record to the final table. If a record with the same ID already exists, we append the current pair to the end of the list (remember, one batch number can occur more once in the file, and we have to collect them all).

	    for i := 0; i < len(records)-2; i++ {
	      v, err := strconv.ParseInt(records[i], 10, 64)
	      if err == nil && v >= 20000000000 && v < 29000000000 {
	        id := records[i]
	        // A map access to a non-existent entry returns the zero value.
	        // The zero value of a slice behaves as an empty slice.
	        table[id] = make([]BatchUpdate, 0)
	        table[id] = append(table[id], BatchUpdate{records[i+1], records[i+2]})
	        i += 2
	  return table, nil

Now, I have the LoadBatchTable() function returning the contents of the PDF as a Go map. It is time to think about storing this map to Memcache and creating the ability to read it back. In the next lines, I introduce another structure called Table. It contains a timestamp of when the record was created (an UpdateTime field) and a map with the information from the PDF file (a Batches field).

	type Table struct {
	  UpdateTime string
	  Batches    BatchTable

If you are not familiar with memory management in Go, it is important to emphasize some key principles. First, Go is a garbage-collected language, so we don't need to worry about freeing memory. Second, Go has pointers. But pointers in Go are much safer than in C or C++ because they don't provide pointer arithmetic, so the pointer in Go is more like references in C++. Moreover, Go can implicitly dereference pointers when you need to access fields of the pointer's underlying structure. For example, if you have a pointer of the type BatchTable declared as p *BatchTable, you can either access UpdateTime via p->UpdateTime or p.UpdateTime. It offers some syntactic comfort. Third, Go can pass objects of any type either by pointer (that is, reference) or by value, but some built-in types (for instance, maps and slices) are always passed by pointer. For example, a variable of the type BatchTable is always passed by pointer, but of the type Table, by default by value. That is why in the next lines of code, we declare a type of the function return value as a pointer to void deep copying.

The next lines also contain a function reading the table from Memcache. Memcache API allows putting values of either the string type or arbitrary compound types into it. We will use the compound type Table. Go has a built-in mechanism called "Gob" for serializing any variable into an array of bytes. In the next lines, we ask Memcache to read a record named table and de-serialize it as avariable of the Table type. If there is no such record, we assume that Memcache has purged it or nobody read the table since the service started, and we load up the table from the network.

	func ReadTable(c appengine.Context, reload bool) (*Table, error) {
	  var table Table
	  if _, err := memcache.Gob.Get(c, "table", &table); err == memcache.ErrCacheMiss {
	    c.Infof("No [table] record found in cache, [%v]", err)
	    if reload {
	      table = *RefreshTable(c)
	  } else if err != nil {
	    c.Errorf("Unable to read [table] record, [%v]", err)
	    return nil, errors.New("Unable to read a record")
	  c.Infof("Read %d records, updated at [%s]", len(table.Batches), table.UpdateTime)
	  return &table, nil

In the next lines, there is a counterpart function storing the table to Memcache. Again, before storing, Memcache serializes the table into bytes via Gob.

	func StoreTable(c appengine.Context, table *Table) {
	  c.Infof("Storing %d records, updated at [%s]", len(table.Batches), table.UpdateTime)
	  record := &memcache.Item{
	    Key:    "table",
	    Object: table,
	  if err := memcache.Gob.Set(c, record); err != nil {
	    c.Errorf("Unable to store [table] record, [%v]", err)

At this point, I am pretty much done with the core functionality.I am able to load and parse the PDF, then to store our data structure back to the GAE cloud via Memcache.

For the sake convenient debugging, let me implement one more thing. I should have the ability to print out the entire table of batches. With a text template library available in Go, the implementation of such functionality takes a couple lines of code. I define a template of an HTML page representing a table. To understand template syntax, you basically need to know only two things: placeholders and clauses. The placeholders are represented as {{.PlaceHolderName}}. Placeholders will be replaced by a variable (or a field) from the template data source named PlaceHolderName. There is also is a range clause allowing traversal of a map or an array.

	const (
	  PrintTemplate = '
	      table { border-collapse:collapse; }
	      table, th, td { border: 1px solid black; padding: .2em; }
	      td { vertical-align:top; }
	      # records: {{len .Records}}, 
	      updated: {{.LastUpdateTime}},
	      now: {{.Now}},
	      age: {{.Age}},
	      <a href="{{.PdfUrl}}">Original PDF</a>
	     {{range $id, $updates := .Records}}
	           {{range $updates}}
	             {{.Status}}, {{.Date}}

Now let's implement a function printing the table via this template. In the following lines, Iprepare a data source. Note that, despite its native compiling nature, the Go language has a very powerful introspection mechanism. This allows the text template engine to be flexible in terms the data source type. For example, if the data source is a map, the map keys will be become the placeholder names, but if the data source is a structure (as in our case), the template engine will automatically discover the field names and use them as the placeholders. I also parse and execute the template. The output will be sent to the outgoing stream represented by the w variable.

	func Print(w http.ResponseWriter, r *http.Request) {
	  c := appengine.NewContext(r)
	  c.Infof("> Print")
	  defer c.Infof("Print finished")
	  w.Header().Set("Content-Type", "text/html; charset=utf-8")
	  table, err := ReadTable(c, false)
	  if err != nil {
	    fmt.Fprintf(w, "No data")
	  last, _ := time.Parse(time.RFC3339, table.UpdateTime)
	  now := time.Now().UTC()
	  data := struct {
	    PdfUrl                   string
	    LastUpdateTime, Now, Age string
	    Records                  BatchTable
	  template.Must(template.New("Data").Parse(PrintTemplate)).Execute(w, data)

By the way, Print is our first function handling HTTP requests in GAE. Such functions always have two parameters: a request and a response writer. The GAE context must be retrieved from the request.

In the next lines, we have a function handling requests to reload the table from the PDF file.

	func Refresh(w http.ResponseWriter, r *http.Request) {
	  c := appengine.NewContext(r)
	  c.Infof("> Refresh")
	  defer c.Infof("Refresh finished")

Next, we need the function that actually executes the reload. It loads and parses the PDF file, puts the result to the table, and updates the current timestamp.

	func RefreshTable(c appengine.Context) *Table {
	  c.Infof("- Refreshing a table")
	  table := new(Table)
	  table.Batches, _ = LoadBatchTable(c, PdfUrl)
	  table.UpdateTime = time.Now().UTC().Format(time.RFC3339)
	  c.Infof("Storing a table to memcache")
	  StoreTable(c, table)
	  return table

We are approaching the finale. The function GetBatch() implements a data retrieval for a given batch number. First, it reads the table from Memcache, checks whether it is time to refresh the data (our obsolescence time is one hour), and finally gets the record from the table.

	func GetBatch(c appengine.Context, id string) []BatchUpdate {
	  table, err := ReadTable(c, true)
	  if table != nil {
	    return nil
	  gap, _ := time.ParseDuration("1h")
	  last, _ := time.Parse(time.RFC3339, table.UpdateTime)
	  now := time.Now().UTC()
	  c.Infof("Last updated: %s, now %s, gap %s", last.String(), now.String(), gap.String())
	  check := last.Add(gap)
	  c.Infof("Check time %s", check.String())
	  if check.Before(now) {
	    c.Infof("Time to update")
	    table = RefreshTable(c)
	  if record, exists := table.Batches[id]; exists {
	    return record
	  return nil

Now, I'll implement our last HTTP handle for the /batch/ function, and I extract the requested batch number from the URL. Then, after retrieving the batch information, I either report that such a batch doesn't exist or iterate over all the records belonging to the given batch and print them all.

	func Batch(w http.ResponseWriter, r *http.Request) {
	  c := appengine.NewContext(r)
	  w.Header().Set("Content-Type", "text/plain; charset=utf-8")
	  parts := strings.Split(r.URL.Path, "/")
	  id := parts[2]
	  c.Infof("> Batch: [%s]", id)
	  defer c.Infof("Batch done")
	  updates := GetBatch(c, id)
	  if updates == nil {
	    fmt.Fprintf(w, "Batch not found")
	  } else {
	    for _, update := range updates {
	      fmt.Fprintf(w, "%s\n%s\n\n", update.Status, update.Date)
	  c.Infof("%v", updates)

That's it! Now we have the full implementation of our GAE-driven Web service. Let's save it as a file called usvisa.go and give it a whirl!

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.