Differences Between Paradigms
Figure 1 compares the success rates between different development paradigms. For each paradigm we asked people what they believed to be their average success rate, giving them ranges such as 91% to 100%, 81% to 90%, and so on. We also allowed people to indicate that that type of approach wasn't applicable for them. We then calculated the weighted average for the responses, with the middle value (95%, 85%, …) used for the calculation. We found that project teams following an iterative process such as Rational Unified Process (RUP) did best with an average 71% success rate. Second best were agile project teams, perhaps following Scrum or Extreme Programming (XP), with an average 70% success rate (statistically the same result as iterative teams). Third place were traditional project teams following a serial approach to development, perhaps based on the V model, with an average 66% success rate. Finally, ad-hoc teams which didn't follow a defined process had a success rate of 62%.
Once again, I'd like to make a few observations. It isn't a surprise that ad-hoc approaches have the lowest success rate, but it is interesting that there isn't a significant difference between ad-hoc and traditional approaches. I was a bit surprised to see that iterative and agile approaches had the same level of success, I was expect agile to be a bit more successful due to its greater focus on collaboration and quality over iterative approaches. This leads me to believe that the greater levels of discipline that we see on agile/iterative teams, in combination with the reduced feedback cycle of those approaches, help to increase the chance of success. Also, other surveys that I've done within the agile community, see www.ambysoft.com/surveys/ for details, leads me to believe that there isn't as much of a difference in practice between iterative and agile approaches as they claim -- there's a lot of talk about "really cool" agile techniques on agile lists, but many of those techniques have been common on iterative teams long before the term agile was popularized and for some of the more difficult practices such as test-driven development (TDD) there appears to be a lot more talk than actual practice.
Distribution Increases Risk
You can also see in Figure 1 that we explored the effect of team distribution on the success rate. For each paradigm we considered three levels of distribution: co-located teams where everyone, including primary stakeholders, are in the same room; near-located teams where some people may be in cubes, or on different floors, or different buildings, or working from home BUT everyone is in the same geographic area; and far-located teams where some people are at such a distance that they would need to take a plane to attend a physical meeting of the entire team. For all four paradigms the success rate went down as the level of distribution went up. Furthermore, in all levels of distribution the effectiveness of each paradigm remained relative to the others -- iterative was always slightly higher than agile, which in turn was always noticeably more effective than traditional, which in turn was more effective than ad-hoc.
The implication of these results is that you should first strive to avoid distributing your development teams whenever possible, just by having people working on different floors you increase your risk, and if you can't do that then you need to adopt more effective ways of working. In The Distributed Agile Team, I wrote about strategies for effective distributed development on agile teams, although much of the advice is applicable for non-agile teams. Distributed teams suffer from increased communication risk, so adopting strategies such as getting key members of the team together physically at the beginning of the project to build relationships between them, having key people travel between locations throughout the project, and reducing your reliance on documentation (the worst way to communicate information) is vital to your success. Organizing your team structure around the architecture, instead of around job function, is also critical -- too many development teams are hobbled by the decision to have the modelers in one location, the programmers in another, and the testers in another. Finally, adopting modern development tools which support the needs of distributed development, tools from even just a year or two ago will be oriented towards co-located teams, can also increase your chance of success.
