In This Issue:
- Agile and Traditional Governance
- Hot Links
Agile and Traditional Governance
The other night I gave a Scaling Agile presentation to the Washington DC chapters of Agile Project Leadership Network (APLN) and the International Association of Software Architects (IASA). Towards the end of the presentation I overviewed the practices of Lean Development Governance and someone asked a question about how agile projects would fit into an existing IT governance structure. This is a fairly common question that I get from time to time and it struck me that it's something I should write about in a bit more detail. So here goes.
The easy answer is "yes", agile software development projects can very likely be shoe-horned into your existing IT governance program. The not-so-easy answer is this likely isn't a good idea because your existing IT governance program is probably a mess that's potentially doing more damage than good. Worse yet, you might be so overwhelmed with rhetoric from the traditional governance folks that you aren't able to even observe the problems that your governance program is causing you. As I've written previously, traditional governance programs are often based on command-and-control strategies which often do little more than layer yet more bureaucracy into your IT department, thereby increasing risk. Many IT governance programs are effectively out of control in practice due (ironically) to a lack of governance: a favorite question that I like to ask senior managers is how do they governing their governance programs, to which I often get blank stares. Effective governance, which agile development teams should step up and demand from their organizations, is based on enablement, motivation, and collaboration " governance should help, not hinder, development teams.
In this gentleman's case his organization has an existing Earned Value Management (EVM) approach to monitoring project teams. Unfortunately EVM is a hot button topic for me and I suspect I subjected the audience to a bit more ranting than they wanted about EVM's inherent evils. Specifically, EVM doesn't actually measure earned value at all but instead tracks cost and time expended against the original plan, so in effect EVM is little more than a euphemism. The real problem is the assumption that the original plan was accurate, an assumption which invariably proves to be false, so in practice EVM effectively measures your progress against a work of fiction. All is not lost because smart managers know that they need to update their schedules and estimates throughout a project to reflect what is actually occurring. So, in this situation under an EVM regime you would then update your EVM calculations to reflect your new plan, enabling you to show that you're pretty much on track if you update on a frequent basis. The implication is that either EVM is an expensive way to gauge how poor your up front planning ability is (something you probably know already), a way for management to force projects to stick to an untenable plan (which would effectively be poor governance), or an expensive way to show that you're sticking to your evolving plan.
EVM also blinds management to a very big risk inherent in waterfall approaches, the fact that true value isn't delivered by traditional development teams until very late in the lifecycle. To see why this is true, consider the following example. A traditional team and an agile team are each part way through their respective projects, and both teams are claiming 70 percent earned value. The traditional team is a little more than half-way through the construction phase and the agile team is into the 8th of 12 planned iterations. The traditional team has some of its code working but still has a lot of programming to do and has yet to begin serious testing. The agile team has been implementing requirements in priority order and producing potentially shippable software each iteration. For whatever reasons the funding gets pulled from each project team and each are given two weeks of clean up time after which they are expected to deploy their system into production. The traditional team isn't in a position to ship anything because the system is only partially built and effectively untested. The agile team has a bit more than 2/3 of the most important functionality implemented and tested. In the end the traditional team hadn't earned any value at all, regardless of what they'd reported to senior management, whereas the agile team did in fact earn the value they were reporting (granted, 2/3 of the functionality might not be enough, but at least they have a viable option of shipping something). The implication is that an EVM-based approach papered over the fact that traditional approaches don't produce real value, which in the case of software development is the production of working software, until the very end of the project.
A more insidious problem is in organizations that have a gated governance strategy based on the traditional waterfall approach. The basic idea is that there is some sort of up-front planning stage where the team needs to do initial requirements, perform some sort of feasibility analysis, and then put together a project plan for the rest of the effort. The next gate is usually requires some sort of architecture or detailed design to be developed and then signed off, and so on. At a high-level these sound like fairly decent governance gates, but in practice they tend to motivate very bad behavior on the part of development teams (more on this in a minute).
My experience is that you'd rather have gates defined around stakeholder concurrence and proving the architecture works through building a working skeleton of the system (along the lines of what's suggested by the Unified Process). These gates are focused on reducing major risks on your project, the stakeholders not agreeing on the goals of the project and the technology not working respectively. The difference is in the level of documentation and bureaucracy that you take on. With the Unified Process approach you don't need to baseline the requirements to come to stakeholder concurrence (two things which are orthogonal to each other), nor do you need to baseline the detailed design to sign off on the architecture (also two orthogonal concepts).
The problem arises when management demands that the project teams provide detailed project plans and estimates early in the project, and expect them to stick to them. Or to produce detailed designs before coding to those designs begins. Organizations that try to fit agile development into waterfall governance frameworks such as this typically have a very narrow view of agile being only applicable to software construction. The reality is that agile approaches are applicable to the entire software development lifecycle, not just construction. By forcing your agile teams to conform to your waterfall governance structure you still take on the risks of serial approaches, including:
- Lower stakeholder satisfaction. Overly detailed specification early in the lifecycle, referred to as "Big Requirements Up Front" (BRUF), results in delivering a lot of software that people don't actually want.
- Lower quality. Early, detailed specification of the design results in systems which should work on paper but often run into significant problem months or years later during construction when the team discovers problems that the architects didn't think through. Furthermore, hand-offs between groups of people increases the defects being injected into the product, and the collaborative and iterative nature of agile has far fewer hand-offs than traditional approaches.
- Lower returns. Total value of ownership (TVO) is reduced because the longer time required of serial approaches pushes out the period during which you gain value from the release of the system. Also, serial approaches typically push significant testing efforts to the end of the development lifecycle, increasing the average cost to fix the defects once they are found (the average cost of fixing defects has been shown to rise exponentially the longer the time between when they are injected and subsequently found).
In summary, effective governance is very important to the success of software development teams, something that is true regardless of your development paradigm. Although traditional approaches to governance may work well for traditional development teams, and please note the use of the word "may". they often prove detrimental to agile development teams. When adopting agile approaches within your organization I highly suggest that you take the time to rethink your approach to IT governance.
My October 2008 newsletter Why Developers Should Demand Effective IT Governance (Yet Have Little Hope of Getting It) examines the issues surrounding traditional strategies for IT governance and argues for leaner, more agile strategies.
The IBM Whitepaper Lean Development Governance by myself and Per Kroll overviews a collection of practice for effective governance of development projects.
My blog posting Scaling Agile with Risk-Based Milestones shows how to apply the Unified Process's milestone review points to effectively govern agile projects.
My August 2008 column, Fixed-Price Software Development Unethical? examines the impact of doing detailed "accurate" up-front estimates on development projects.
My September 2007 column Agile on a Fixed Budgetdescribes strategies for running agile projects for all combinations of fixing, or not fixing, the schedule, scope, and budget. So, if you're going to do fixed price you should at least be smart about it.
My May 2008 newsletter Questioning Earned Value Management on IT Projects overviews why EVM proves to be a rather poor approach to monitoring IT projects. To be fair the traditional folks have few decent options available to them due to the inherent risks associated with serial approaches, but the agile community can do much better.
My article The Agile Software Development Lifecycle (SDLC) overviews the full agile development lifecycle, giving you a better understanding of how agile strategies apply to the full lifecycle and not just construction.
The article Examining the Big Requirements Up Front (BRUF) Approach overviews the evidence showing that writing detailed requirements documents early in the lifecycle leads to significant levels of waste on software development projects.
The Agile Project Leadership Network (APLN) homepage is here.
The International Association of Software Architects (IASA) home page is here.
The article Why Agile Software Development Techniques Work: Improved Feedback discusses the impact of adopting agile techniques which have a shorter feedback cycle than traditional techniques.
My Agility@Scale blog discusses strategies for adopting and applying agile strategies in the complex environments.