Rethinking Documentation
Documentation is as much a part of the system as the working software itself. Your goal should be to develop documentation that is sufficient for the needs of its audience, and to do that you must work closely with your stakeholders to understand their actual requirements. I believe that this is where agilists truly differ from their traditionalist counterparts: Agilists treat documentation as a stakeholder requirement, not as something dictated by the process. Because documentation is a requirement, it is something that should be estimated by the development team and prioritized by the stakeholders. Because the resources that you have to invest in an IT project are finite, and because it's your stakeholder's money being spent, they should decide which documentation will be created and how comprehensive it needs to be.
Because your stakeholders are a varied bunch, they consist of end users, business management, IT management, operations staff, support staff, enterprise architects, and many more, chances are pretty good that most someone will identify the need for a document. But let's assume that they forgot to ask for system overview documentation. Does that mean that you don't create it? No, of course not. What you do is suggest to your stakeholders the need for such documentation, justify why it's important, estimate the total cost of ownership (TCO) of creating and maintaining it, and then let them decide whether the wish to invest in the documentation.
This is a very different approach than what traditional teams typically take. Traditional software processes define which documents need to be created, will often provide detailed guidelines and templates to help ensure completeness, will indicate when the documentation should be created, and who should receive it. The underlying assumption is that the stakeholders, the people who were smart enough to earn the money, aren't smart enough to determine how to spend the money and therefore those decisions are taken away from them in traditional processes. Not only is this incredibly arrogant, it is also ripe for abuse—hence the overly bureaucratic and documentation-heavy processes that we see in many organizations.
Many organizations justify their burdensome documentation practices on industry regulations such as the Food and Drug Administration (FDA) regulations, the Sarbanes-Oxley (SOX) act, or the Basel-II regulations. I've worked in organizations that have had these regulatory requirements inflicted upon them and I can safely assure you that it is possible to remain agile. The first, and more important, step is to actually read the regulations. Nowhere in them does it say that you need to be ineffective and wasteful, yet if you allow the paper-pushers within your organization to interpret the regulations (and doesn't it seem that they're always lining up to do so?) then inevitably you'll find yourself in a bureaucratic morass. The second step is to be actively involved with interpreting the regulations to ensure that your resulting process is as streamlined as possible yet still in compliance. Third, be prepared to evolve your process to reflect new versions of the regulations as they evolve over time. For example it is likely that SOX will be simplified soon, and I think that it will be interesting to see how many organizations choose to tear down the political empires that were justified based solely on SOX.
