Test-driven Development, Expect, and Systems Administration

There's a better way to do systems administration.

That's my belief, anyway. To control at least part of the crisis-driven noise that so often plagues systems administration jobs, I favor "test-driven development" (TDD) practices. The first installment in this series introduced the Expect tool and language. This installment shows how Expect models solutions for tests that matter to systems administrators.

This article has been a particular challenge for me — it involves not only several different crucial ideas, but different kinds of ideas. Precision about the role of Expect is an example of this difficulty.

Where does Expect fit?

As the previous article explained, I find Expect nearly indispensable for rational Unix systems administration. Almost every week, I come across a task that Expect solves in minutes, where other approaches would require hours. It consistently pays big returns for the time I invest with it.

Testing happens to be one of Expect's strengths. DejaGnu is a well-known testing framework, arguably the most important such framework in the history of computing. Still going strong in its second decade, DejaGnu is most famous for its role in ensuring the correct function of the GNU gcc compiler on which Linux and so many other projects rely. DejaGnu is based on Expect. This article will provide a few samples of working Expect code you might use to illustrate the role testing can play.

At the same time, though, the concepts behind Expect, and TDD more broadly, are even more important than their expression in any one language or tool. Even if you know you'll never write Expect scripts, you should keep reading. TDD is that important — besides, the third and final installment in this series will show how to do Expect-style jobs without Expect.

TDD practices aren't new. While they've been around from the origins of computing, every information technology (IT) generation seems to need to rediscover and rename it. Moreover, systems administration has never accepted TDD with even a fraction of the enthusiasm I think it deserves.

A little example

Part of the reason probably is because it's hard to teach TDD from a systems administration viewpoint. Our typical chores are often site-specific, tie in to installed infrastructure, and vary a great deal even from one flavor of Unix to another. These realities limit the clarity of all but the simplest demonstrations. Let's start, therefore, with a very simple case.

When someone leaves your workgroup, his or her id needs to be disabled. That policy requirement might involve a variety of implementation details: maybe LDAP updates through a Web interface, maybe a simple deluser(8) on one machine, or any of several other possibilities. The first step in TDD is to operationalize the requirement unambiguously — to make it perfectly clear what we mean by "disabled". For this example, we'll take it as, "returns 'Unknown id ...' on an attempt to su". Try this for yourself:

# su nemo
  Unknown id: nemo

 #!/usr/local/bin/expect
    package require tcltest
    namespace import ::tcltest::test
    
    test tcltest-0 {This simple illustration--just
		     for practice--of how tcltest works
		     uses arithmetic everyone understand;
		     it asserts that "8" is the sum of
		     "3 + 5"} {
      expr 3 + 5
    } 8
    
    
    test admin-user-0 {'nemo' is known not to be
         a valid id, so this test confirms the
         response to an id known invalid} {
      catch {exec su nemo > /dev/null} result 
      set result
    } {Unknown id: nemo}
    
    
      # Package the exercise to make it easier
      #    to use.
    proc su_response id {
      catch {exec su $id > /dev/null} result
      return $result
    }
    
    
    test admin-user-1 {This just rewrites
        admin-user-0 in terms of a convenience
        procedure} {
      su_response nemo
    } {Unknown id: nemo}
    
    
    test admin-user-2 {This is a *real* test,
        in that it will fail until procedures
        have been implemented for creation and
        removal of users} {
        # admin-create and admin-remove need
        # to be written locally.  Notice that
        # they are fully general, in principle;
        # they might combine administrative
        # action "by hand" and automation.
      admin-create test_user
      admin-remove test_user
      su_response test_user
    } {Unknown id: test_user}

If you do so, the first three tests will pass, but Expect will complain that test "admin-user-2" fails, and explain briefly how it fails.

What's the point of this apparent detour through scripting? It transforms the English-language policy requirement, "... id needs to be disabled", into a definite coding goal — make admin-user-2 pass. That's how top developers work; they have clear goals, and the programs they write or configurations they set are steps to those goals. We can do much the same in systems administrations — establish clear targets for ourselves, and leverage coding techniques to help us advance toward those targets.

Assets

While that's an advantage in itself, the advantage multiplies when we fully automate such tests (for more depth on this subject, see the book review on UnixReview.com). Suppose you have a system that allows users to enter "help tickets" through email, and that you can access the ticket collection through a Web interface. Expect knows email and the Web; it takes only a few lines to write tests that verify entry of a ticket through email, and its appearance in a Web-based report. Other tests might check that the system properly closes out completed tickets, or cross-references related items.

And when it's all automated, it's much easier to maintain. Your network can run a full battery of tests on itself every day, during slack hours, to help detect surprises (what happens when the mail server is off-line? What if /tmp fills up on the Web server? and so on) early, when they're easy to localize, before they become emergencies.

Automation of this sort also "depersonalizes" administrative work. Conversations of the "I thought you said you checked it. I did, but you didn't tell me we had to ..." sort dissipates far too much of the energy of systems administrators. Construction of objective tests, however simple, goes a long way toward improving communication and highlighting real trouble spots that merit more attention.

Expect's role in all this is to serve as a sort of "existence proof". The important idea is to express specific, testable objectives. While plenty of tools have the potential to do this, I use Expect because:

1. Many systems administators and network managers already are familiar with it;
2. Expect scripts are often readable even to those who don't know the language;
3. Expect is great for dealing with cumbersome interfaces involving passwords, remote access, and serial-line and other connections to specialty hardware;
4. Expect supports the tcltest and DejaGnu testing frameworks, and even makes it easy to parametrize tests written with them;
5. Expect is widely available and very portable across Unix flavors;
6. With the right extensions, Expect even can speak SNMP and CORBA.

You can approach the "test-driven administration" I describe here incrementally: write your first test for a new task whose validation should be particularly easy to automate. Build on that. As time goes on, you'll build up an inventory of "self-monitoring" systems. More of your time can go to long-term planning, experimentation, and development; emergencies should take less. Fewer of your hours will be spent in crisis, and more in design of solutions. Moreover, tests that express requirements objectively also make it easier to pass on responsibilities to co-workers; it's easier to "see for themselves" how things are supposed to work.

What's your experience? Is systems administration in your shop too chaotic to permit automation? Do you have a better way to manage your responsibilities? Do you face specific implementation challenges that you don't know how to solve? Write me and perhaps we can find solutions based on Expect or other tools.

Next month, I'll write about Expect's surprises — things it does that aren't well known, and easier ways to do things for which it's commonly used.