Ed's an EE, PE, and author in Poughkeepsie, NY. Contact him at [email protected] with "Dr Dobbs" in the subject to avoid spam filters.
Eagleson's Law: Any code of your own that you haven't looked at for six or more months, might as well have been written by someone else.
— Found floating on the Internet
Semiconductor memory datasheets provide nearly all the information required for a successful design: control signal sequences and timings, environmental requirements, and even the expected lifetime for those bits under various operating conditions. When you apply a dash of engineering judgment to the specifications, you can be reasonably sure that the device will work correctly.
Conversely, human and institutional memories not only lack datasheets and warranties, but tend to vanish without warning.
Let's take a look at some organic memories, starting inside the skull.
Removed Memories
Did you see Paycheck during the last year because Ben Affleck played an engineer, for the John Woo SFX, or to catch Uma Thurman in the shower? Evidently, none of the above was sufficiently compelling, because the movie was something of a box-office dud. Let's skip through the plot and see what Hollywood thinks of engineering.
The protagonist, one Michael Jennings, is a world-famous "reverse engineer" who freelances for companies that steal IP from competing products. Jennings has his memories of each job erased to avoid repercussions: What he does not know, he cannot tell. Oddly enough, the memory- eradicating machinery plays his vanishing thoughts from a third-person overhead perspective on a high-res color monitor. I don't know about you, but my memories don't work that way. Ohhh, that smooch! Never mind.
We're asked to believe that Jennings goes through his life with gaps of a month or two, rebuilding his expertise from a stack of magazines stored away by his confidant/mentor/agent/personal trainer. It's not clear how often he did these jobs, but they were evidently fairly lucrative. Pity, though, that he never recognized his coworkers at the product-release parties.
During the exit interview following an unprecedented multiyear gig at a college buddy's tech company, he's handed an envelope of trinkets that his prewipe self has agreed to accept in lieu of the expected "eight figure" paycheck. The movie's plot, such as it is, revolves around his gradual discovery that the trinkets help him escape from a succession of crises, culminating in the predictably spectacular destruction of the entire lab where (ah-hah!) he developed a future-viewing time machine.
Evidently, the lab had no data backup, no off-site storage, and all the knowledgeable bad guys get killed. Okay, bad guys might not be too keen on duplicate records and even good guys have trouble with backups. Never mind.
One of the few things I know for sure is that expertise is more than knowledge. As the old saying goes, good judgment comes from experience and experience comes from bad judgment. You build expertise as you go, starting from book learning and adding real-world experiences. Expecting an engineer to maintain world-class expertise without allowing him to learn from successive jobs reveals a deep misunderstanding of what goes on between our ears, not to mention the effects of Moore's Law.
There's actually a medical diagnosis for Jennings's business model: Korsakov's syndrome. A Korsakov's patient presents severe retrograde amnesia that wipes away all recent memories while, paradoxically, leaving cognition and old memories completely intact. Unlike Jennings, however, a Korsakov's patient cannot fix new short-term memories, and thus, has lost the context of his existence.
Not a pleasant thought, is it?
Retrieved Memories
In my July 2004 column, I mentioned that I might be The Prior Artist in a patent lawsuit. The date when we published the project was on record, but the key date occurred when I first reduced a particular algorithm to practice. All I had to do was find all my project records from 20 years ago.
Being that type of guy, and having garnered some experience along the way, I knew the records were in the stack of cardboard boxes we've been toting from house to house over the years. So I confidently pulled out the appropriately labeled box, extracted that year's notebook, and discovered that the pages from the relevant dates were missing.
Given that hint, I recalled preparing a project summary as part of a different lawsuit that Went Away without my having to do anything else. Some concerted rummaging unearthed that summary, but not the notebook pages, in a filing cabinet. Evidently, I'd extracted the pages to compile the summary, made copies, and put the originals in a secure spot. But where?
Fundamental Algorithm: All searches degenerate to linear. When all else fails, you must read sequentially from the beginning of the database until you hit paydirt. That amounted to 10 feet of notebooks and a dozen cardboard boxes of records, assuming that I didn't need to examine any of the hundred-odd cartons of hardware, electronic parts, machine tool parts, and so forth, and so on. I have, as you might expect, a rather large basement office cum machine shop that's chock-full of nifty stuff.
So I searched diligently, but not exhaustively, failed to find the records, and reported back to the guy who contacted me. It was my best effort and I assumed that would be enough.
A few weeks later, one of their lawyers drew the short straw and sat cross-legged on my concrete floor, opening boxes and searching for records. She verified that I'd done a good job on the original search by also not finding the missing pages in any of the expected places. The next step was to search, one by one, every other box in the basement.
To truncate a long story, we found the notebook pages in a cardboard box clearly labeled as containing something else, tucked in a hanging folder neatly tabbed as holding something else, in a bunch of papers unrelated to the folder's tab.
I have no idea why or how the notebook pages wound up with those papers in that folder in that box, but there they were. A secure spot, indeed!
Large organizations generally have a somewhat more formal approach to records maintenance, but we've all seen instances where important records go missing. Nixon's infamous 18.5 minutes of silent tape exemplify mysterious circumstances at work and Arthur Anderson's shredded Enron docs look like malfeasance in action, but, generally, folks just discard old stuff to make room for new stuff without realizing just how vital the old stuff may eventually become.
Fortunately, the persistent rumor that NASA managed to either lose or destroy the plans for the Saturn V booster is an urban legend. The original plans live on microfilm at Marshall Space Flight Center and thousands of cubic feet of related records reside elsewhere. What we don't have is the ability to produce space-qualified, 40-year-old parts using today's technology. Most of the original suppliers are long gone, so any new-old parts would require extensive requalification. Even the best-preserved records are useless without the corresponding infrastructure.
Vanished Memories
If employees are the neurons of an organization, what happens when one cell dies? The results range from trivial to catastrophic, depending on just how much that neuron knew. Even "key person" (formerly "key guy") insurance can't cover the loss of knowledge when someone stars in a fatal accident or simply moves to greener pastures.
Small shops know this problem well, because one person typically carries the entire software architecture and its implementation. As he goes, so goes the company.
Larger shops can afford trendy software methodologies, all of which attempt to get at-least-average results from at-most-average programmers, with the predictable result that they regard headcount as fungible. There's even a saying: "Headcount is interchangeable: You make no difference." However, even programming-in-the-large projects tend to have only a few Chief Architects who actually know what's up with the whole system, making a loss at the top predictably catastrophic.
A recent experience illustrates how easily and quietly the Key Guy problem can arise. Bee, a friend of mine, retired in the early '90s and devoted his time to volunteer work and the many clubs and organizations he'd joined over a long career. He was their de facto "computer guy" and, perforce, wound up with their membership records on his PC.
Being that type of guy (yeah, we got along pretty well), he wrote a comprehensive data-management program to handle all the mailings, renewals, votes, and suchlike. Of course, he wrote everything in C++ and, equally of course, he carried the entire project in his head. Wih, his wife, became the Membership Queen, ruling a collection of data with a program that only she ever ran.
A few months ago, Bee suffered a fatal stroke while repairing his 35-year-old Heathkit garage-door opener (way to go, Bee!). I talked with several people at the memorial service who suddenly realized they had lost more than a friend; they had lost the only guy with access to their membership.
Because Bee's program was not proprietary, if not exactly Open Source, I suggested that each club could find another "computer guy" and simply transplant the program, its source code, and the appropriate data to those machines. That might work as an interim measure, but it was obvious nobody could maintain Bee's code. He had been their "computer guy," no one else had the slightest interest in assuming that mantle, and the prospect of porting C++ code from one compiler to another is a daunting prospect even for well-equipped IT shops.
When I examined the program and the databases, it wasn't clear how Bee had organized things. In fact, I couldn't immediately tell if the membership data resided in a single massive file or if each organization had its own file, because the various files scattered around had considerable overlap. The program lacked a data-export option, so they'd need a script to tear apart fixed-length database records and create something like a CSV file that could be imported into a different database program.
One can argue that C++ isn't the appropriate hammer for the membership roster problem and that there exist myriad database managers much better suited for the task at hand. Bee undoubtedly chose a bespoke solution based on his innate desire to control everything (he was, after all, an engineering bear), the knowledge that clubs typically don't have much interest in data management as long as the newsletters go out and the renewal checks come in, plus his ability to make things work.
If you're the Key Guy, start training an understudy. If your organization depends on a Key Guy (or two or three), think about What Will Happen When...
Recovered Memories
Even if you don't lose a Key Guy, the long-term effect of employee turnover can erode an organization's collective memory. The numbers I've seen range from 10-35 percent annual turnover, which probably shows how little hard data actually exists. A project with 25 percent annual turnover will have only a quarter of the original staff left in five years, at least if they're not simply churning the same slots. Just 15 percent turnover replaces half the staff in that time.
Eventually, despite everyone's best efforts, the organization will forget both the grubby details and the overall architecture that once controlled the design. Instead of writing the least possible code for a given change, you must graft entire subsystems onto the existing program, because the API doc no longer matches the actual code. That way lies madness.
Business mergers and acquisitions also sever programs from the people who once understood them; after a few such transactions there may be nobody left who knows how the code works, let alone anyone who can fix it. Worse, the existing documentation probably doesn't represent the as-built code, so you can't discover what you have without actually analyzing the code.
I recently read several whitepapers and presentations from Klocwork (yes, that's kloc, as in Kilo Lines Of Code) describing their Architecture Extraction, Excavation, or Archeology process, which attempts to reconstruct those high-level design documents. Starting with a project's entire code base, a combination of automated and manual refactoring extracts the underlying semantics of the system in lumps large enough for analysis. The final result is a Summary Model that resembles a UML model with direct links to the preexisting code.
Unlike an outdated architectural model that might still be associated with the project, Klocwork asserts that models excavated from the actual code base have a compelling advantage: They represent what's actually in place, not what was once intended.
You'll probably find a bunch of cruft lurking in that old code: Mysterious routines that make no sense whatsoever. One Klocwork presentation mentions 10 percent dead code in a 10-MLOC project. Yes, that's a million lines of ballast. Think about it, using any numbers you prefer for the cost and schedule of a megaLOC project. Of course, the code didn't actually work, so that might reduce the actual cost.
Klocwork may offer a larger set of hammers than you really need, if only because most embedded projects remain fairly small. Some automated help can save the day when you really must figure out what that huge project actually does, but the next step may be a redesign and rewrite. Perhaps you can regard the existing code as a learning experience?
Better still: Never forget!
Reentry Checklist
Needless to say, those project notebook pages are now in a very well-labeled, entirely separate box, with digital copies in my compulsive backup rotation, and, yes, I have off-site storage. That certainly means I'll get subpoenaed for some other project.
The original version of Paycheck appears in Paycheck and other Classic Short Stories by Philip K. Dick, the first of five volumes. The story's rather quaint feel comes from its 1952 copyright, but you'll find it far darker and more thought provoking than the movie. Jennings is, in fact, not a particularly nice guy.
The Official Paycheck Movie Site is http://www.paycheckmovie.com/, which may require a trip to http://www.mplayerhq.hu/ for a Linux media player that can handle Quicktime. Adding the DVD to your NetFlix queue is probably better than buying this turkey.
William Gibson touched on Korsakov's syndrome in Mona Lisa Overdrive. Slick Henry lost his three years as a punishment for grand theft auto, though, which may be prescient.
Oliver Sacks sympathetically describes many grim neurologic disorders in The Man Who Mistook His Wife for a Hat. The chapters "The Lost Mariner" and "A Matter of Identity" explore the disturbing consequences of Korsakov's syndrome in real patients.
More on the Saturn V at http://en.wikipedia.org/wiki/Saturn_V, including links to the "lost plans" urban legend and its rebuttal.
Klocwork is at http://www.klocwork.com/, another site with gratuitous Flash and drop-down menus that assume entirely too much about your browser.
DDJ
