|
Ed does a postmortem report on those embedded systems we all love to hate—electronic voting machines.
January 11, 2007
URL:http://www.drdobbs.com/embedded-systems/root-the-vote-wetware/196900208
Ed is an EE, an inactive PE, and author in Poughkeepsie, NY. Contact him at ed [email protected] with "Dr Dobbs" in the subject to avoid spam filters.
The best laid schemes o' mice and men
Gang aft a-gley,
And lea'v us nought but grief and pain,
For promised joy.
—Robert Burns
Electronic voting machines are, at heart, devices that present questions and tally the answers. As with all gadgets, when everything works, the final totals correctly represent the machine's inputs every time. Unfortunately, real life rarely works out so nicely.
Much of the discussion about electronic voting deals with how to prevent hardware or software corruption, yet it seems to me that a more serious problem lies in the organizations surrounding the voting booth. While you can fix hardware problems with a forklift and correct software errors with Yet Another Version, those pesky people problems may just be unfixable.
I've presented postmortem reports on various NASA spacecraft, examined automobile recall info, and crawled through other projects to show how good intentions go bad. It turns out that Cuyahoga County, Ohio, used Diebold electronic voting machines in May 2006, an election they describe as "at best problematic and at worst a disaster." The County appointed a panel, which produced a comprehensive Final Report that should serve as a red alert to anyone who believes the solution to electronic voting involves fixing hardware and software problems.
Last month, I discussed some rules for evaluating hardware and software proposals. This time, let's look at what you might expect from the wetware running the show. Italic sentences come directly from the Cuyahoga Report.
You've probably noticed that folks outside your specialty don't quite understand what you do for a living and you can't clarify it for them, either. Indeed, even your parents and children probably lack a good appreciation for your work. The converse is also true, but you might have a harder time appreciating how little you understand what somebody else is up to.
There's a simple reason for that: People concentrate on what's important to them, not what's important to anybody else. For most of human history that didn't make much difference, because everybody did pretty much the same things and everyone's knowledge overlapped.
The process of running an election depends largely on unpaid (or barely paid) volunteers, organized and directed by a sometimes skeletal staff of government employees with other jobs during much of the year. That worked well in the days of paper ballots stuffed into wooden boxes, ran into trouble with mechanical voting machines, and hard-crashed with the advent of computer-based voting technology. Elections happen at most a few times a year, far too infrequently to reinforce the skills required for the job. Verifying signatures, directing voters, and other low-tech parts of the process aren't difficult, but operating and troubleshooting any computer-based system requires well-practiced skills.
Poll workers reported a large number of broken machines...Reasons why these machines couldn't be used included: machine malfunctioned, the machine froze and could not be reset, the printer failed, or the printer was missing and the machine could not be used...There were extensive reports of difficulties zeroing out the machines, either due to lack of training, malfunctioning of the accumulator machine, or because memory cards were placed in the wrong machines.
The Report suggests that, in addition to verifying that the machines are functional and correctly configured before Election Day, the poll workers must be fully trained how to set the DREs up, basic troubleshooting, and on the accumulation process.
To a large extent, poll workers come from the ranks of retirees, if only because they're the only ones with sufficient free time for what can be a 15-hour day. Whether it's reasonable to expect poll workers to put in that much uninterrupted time is another matter: Here in Dutchess County, the Board of Elections strongly recommends that inspectors not leave their assigned polling places for meals or rest breaks, let alone to vote in their own precincts.
My wife has been a Dutchess County Election Inspector for the last several years and reports that, after five elections, she's just now comfortable with mechanical lever machines and the overall voting process. That puts her well ahead of several less-technically inclined compatriots, who depend on her for answers. She laconically describes the training as "inadequate."
In general, most ordinary folks, let alone retirees, lack the qualifications to perform even rudimentary troubleshooting of computer-based systems. If they were qualified, then you wouldn't get nearly as many desperate phone calls from your friends and relations asking for help with their PCs, nor would the major computer manufacturers have such trouble providing tech support.
In summary, mandating electronic voting systems, even those with a paper audit trail, requires far more knowledge from poll workers than can be reasonably expected given the low duty cycle and high technical content of the job.
Ten-year-lifetime lithium batteries and unbreakable LEDs have largely vanquished the dead-flashlight problem, but in general, equipment you don't use regularly probably won't work when you need it. A cell phone left untouched for a month has a dead battery, your house has at least one problem after a long vacation, and your snow tires always need air after Thanksgiving.
The Cuyahoga poll workers tried to get their jobs done under difficult conditions, but sometimes their efforts to cope only made things worse. The Diebold response to the Report summarizes one situation:
Poll workers in various locations apparently pulled memory cards from one touch screen unit and placed that memory card into another touch screen unit. However, they did not also remove the respective VVPAT [Voter Verified Paper Audit Trail] paper tape and place it into the second unit. Clearly, removing a memory card from one unit and placing it into another without also relocating the VVPAT records will account for a discrepancy when those results are compared.
That may be obvious in retrospect, but the situation was more complex. Although the memory cards contain the entire voting record for a particular machine, some DRE units were not marked with a number that corresponded to the memory card. Some cards were moved when that error was discovered, others were moved from one failed machine to another, and still others were lost, replaced, found, and then swapped back in, all to get enough machines working on Election Day.
Pop Quiz: What do you do when a USB Flash drive doesn't work?
Most likely, you'll try it in another port, then on another machine, to see if things improve. What if plugging a memory card into the wrong system invalidated its contents or wiped out the system, but only under certain circumstances? Would you remember that in the heat of the crisis, early on Election Day morning, with lines of voters wondering what's wrong with you?
The VVPAT paper tape isn't readily portable from one machine to another, particularly in mid-roll, and as nearly as I can tell, you'd have to dismantle the printer to extract the complete tape. Tearing the audit trail tape in mid-election probably isn't covered in the procedures and could run afoul of election rules.
A number of DRE units crashed, froze, or malfunctioned during boot-up or use on Election Day, an unknown number of which were returned to service without further investigation.
Each debugging decision and ad-hoc workaround makes sense at the time, can be criticized based on a lack of overall system knowledge, and is the sort of thing that anyone would do to solve the immediate problem without considering its overall effect.
The election officials did provide a voluminous manual. Alas, [t]he final manual provided for use on Election Day contained substantial errors which affected Booth Official's ability to set up and configure voting machines. Anyone who has created documentation can sympathize, but that's little consolation.
Poll workers, being volunteers, also tend to be both highly motivated and dedicated. It's unreasonable to expect them to not fix whatever gets in the way of doing their jobs; indeed, you couldn't ask for better employees.
The trick is ensuring that the rest of the system can keep up with them, even in the face of unexpected failures and the inevitable confusion of real life. Fielding a system that works perfectly under laboratory conditions isn't acceptable; it must also work when used by people who don't understand all its undocumented nuances and take sensible, if sometimes counterproductive, action when (they think it's) needed.
One conspicuous failing, mentioned throughout the Report, was inadequate training for poll workers, employees, and other participants. If only they'd gotten better training, we're told, perhaps the outcome would have been better: fewer mistakes/less confusion/more serenity. While that may be true to some extent, I believe better training cannot solve electronic voting's problems.
There's a subtle, often overlooked, distinction between training and education. The American Heritage Dictionary defines "training" as, "to make proficient with specialized instruction and practice." The dictionary.com definition of "education" is, "the act or process of imparting or acquiring general knowledge, developing the powers of reasoning and judgment, and generally of preparing oneself...intellectually for mature life."
When you're trained, you can recognize specific situations and apply a stereotyped response based on your training: If it's bleeding, apply a bandage. If it's really bleeding, apply a tourniquet and call for backup.
Conversely, education provides deeper background knowledge so that you can not only recognize a situation, but also reason toward a conclusion that depends on both theory and the situation at hand. Your reaction to a given problem won't be as rapid as that of a well-trained person, but you can handle more different situations with appropriate responses: If 10,000 are bleeding, begin triage.
Traditional polling duties, those involving pencils and paper forms, required a very limited set of skills, were easily amenable to rapid training, and were well-suited for low-duty-cycle volunteers. Electronic voting should produce identical vote tallies, but requires a radically different skill set.
As anyone who's done remote tech support, perhaps for a friend with a recalcitrant Windows box, can attest, the symptoms of a failure often have little relation to the actual cause. That's part of the dissatisfaction produced by the well-scripted folks on the other end of manufacturer tech-support lines: The solution always boils down to "reinstall Windows," simply because that's guaranteed to suppress the problem at hand. Should reinstallation blow away your data, well, that's a different problem and not one for which they take any responsibility.
Polling workers cannot be educated to the level of computer-savvy competence required to diagnose and treat the types of failures we've all become accustomed to, if only because most of them lack the background (and desire!) to attain geekhood. The alternative, training them to recognize a failure and call for backup, simply does not scale well to typical elections, as evidenced by the Report's findings.
Poll worker calls to the [support center] were not answered by knowledgeable officials...The calls...were not directed to the proper authorities who could remedy the complaints.
If the only possible response to a "machine down" situation is calling for backup, then the number of trained (if that's the right word) technicians becomes the limiting factor. It's unreasonable to expect adequate and timely backup across an entire election district, particularly during a national election. Remember that elections cannot be rerun the next day; any downtime affects the outcome.
Basically, the overall system must not transform simple, localized failures into widespread problems. As one worker put it, "...I've worked a good many years, but they have to...make it simpler for those people that are not real computer savvy."
The Report concludes that, while there's no evidence of malicious vote tampering, the lack of control over memory cards and machines makes it impossible to be certain. As I said last month, deliberate vote tampering lies in the future, after we have deployed sufficient standardized machines to form a unified target.
An article in nytimes.com describes the peculiar relationship of the Venezuelan government to Sequoia Voting Systems, a Diebold competitor in the U.S. electronic voting machine market. Smartmatic, a company with no electronic voting machine experience before the Venezuelan government tapped it to provide machines for their 2004 elections, now owns Sequoia. The owners of Smartmatic also own a smaller, equally inexperienced, company that joined it to provide the Venezuelan machines, with that government owning 28 percent of the second company. I do not understand who owns what at this point, other than that Smartmatic bought Sequoia using the proceeds of their Venezuelan contract.
As with Cuyahoga County's elections, there is no evidence of foul play and Sequoia's machines have been through all the usual certifications. It should be obvious from the results to date, however, that certification has little effect on the reliability of the machines, their suitability for the job, or the adequacy of the infrastructure supporting them.
The election workers at each polling place can verify that voters are registered and that they cast their votes properly. A careful balance of Democrat and Republican workers prevents undue influence from either party. Unfortunately, those workers can neither prevent malicious software from affecting the results nor verify that such tampering has occurred.
Worse, because external certification inspections are essentially black-box tests, there's no way to examine the actual code base. Both Diebold and Sequoia claim to have good code (as do all vendors of all products!), but voting is one application area where those assurances simply aren't sufficient.
The Report observes that ...the problems experienced left the impression that machine tampering was a distinct possibility. Ensuring that the machines work is a necessary part of using electronic voting, but it's not sufficient. The entire election process, from voting through tabulation to presentation, must be open to inspection and verification. Any part of the operation labeled, "Trust me, this is OK!" marks the spot where tampering will occur.
I've heard from several folks with proposed technological fixes for various voting machine problems, all of which miss the mark. It seems to me that, if the source code and circuitry were open to public inspection, the actual machinery wouldn't matter.
The real problems lie in the election infrastructure, which must not only not become swamped by tech failures or corrupted by outside influences, but must also retain visible and verifiable integrity in the face of those influences.
If you can help accomplish that, perhaps we can be smart enough to avoid some serious grief and pain. One can hope, anyway.
The relevant reports from Cuyahoga County are available through bocc.cuyahogacounty.us under "Investigation Reports" near the bottom of the page, along with Diebold's response. Evidently, planning was not their strong suit.
Avi Rubin's blog at avi-rubin.blogspot.com covers many of the problems with voting, both electronic and manual. He's been a poll worker, too, so his knowledge runs the gamut. Verified Voting is at www .verifiedvoting.org.
Get refereed definitions at dictionary.reference.com. Newer words, slang, and jargon appear at www.urbandictionary.com.
The nytimes.com article is at www.nytimes.com/2006/10/29/washington/ 29ballot.html. You'll need registration and perhaps paid TimesSelect membership.
Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.