I'll make a confession: I'm not organizationally gifted. This was painfully clear recently when I frittered away frustrating moments of a busy day searching for a document that I needed now! It occurred to me that, had I organized and versioned the document more appropriately, I would surely have what I needed -- without delay and without the heart palpitations.
It was during this small and mundane moment that I got thinking about the state of enterprise IT today. Most IT shops look an awful lot like my desktop. Of course, I'm only slinging documents. IT systems are a different deal entirely.
For IT, in the haste to meet deadlines, a lot is left to chance. Many best practices are traded for paths of least resistance. Systems are cobbled together using more art than science; what software is used -- sources, versions, dependencies -- remain unclear. So, when the system finally makes it to production, it immediately falls down -- or threatens to do so just as soon as you fiddle with anything. For many organizations, update processes are as unsettling as a surgeon with astigmatism.
A profound lack of transparency ensures that, when change occurs, bad things happen. It's no wonder that 80 percent of IT outages are attributable to system updates -- it's the Law of Unintended Consequences enacted, ratified and applied in full force.
You can trace the origin of this Law to the way systems are constructed today:
- Developers grab whichever IT-issued platform is available to build an application.
- They throw it over the wall to QA, who recreates the system based on whichever version of the platform they have laying around; they resolve undocumented dependencies, tweak, tune and fiddle until it's working.
- They certify the system and pass it to the production teams who recreate the system, again, based on their latest version of the platform.
During each of these handoffs, software versions and system configurations drift. Dependencies are missed, found, resolved, poorly documented and, consequently, missed again. Deployments fail, the system cycles back to development and test for hot fixes and recertification. Finally, the production team gets the system to run. Exactly what gets deployed is anyone's guess.
When it's time to update the system, the scenario becomes even more gruesome -- resulting in one of three certifiably dysfunctional behaviors:
- Updates are deferred (putting compliance and performance at risk).
- Changes are made blindly (leading to outages and job retention issues).
- Changes are tested extensively (slowing IT and adding significant cost).
Of course, there's a better way. It's an approach based on the principles of knowledge management which brings us back to my document issue.
Just as I need a consistent way to organize and version my documents, IT needs to do the same for all of its system artifacts. The solution begins with a "definitive software library," which is a fancy ITIL description of the source of record IT organizations must establish for controlled reuse and management of system artifacts across the operational lifecycle.
From this version-controlled software library:
- App dev gets the latest platform against which they build a new application.
- They then check in the complete, dependency resolved system -- application, OS and middleware components -- which is identified by its own version number.
- QA is alerted, and they pull down the very same version and run their test suite.
- Production grabs the certified version and successfully provisions the system.
Dev, test and prod are all accessing the same exact version -- no drift, no confusion.
And once a system is deployed, a version manifest becomes the basis for seamless change. Updates can be quickly matched to system inventories, change impact is clearly documented, and patches and updates can be implemented incrementally. When outages do occur, restoration is as simple as reverting to the previous version. It's a refreshingly simple answer to a complex operational challenge.
So, with that under control, I suppose it's time to look at my document issues