Agile and Iterative Work Better
Figure 2 presents ratings of the four development paradigms against each of the four success factors which we looked into. The scores are weighted averages calculated by the where a Very Effective ranking was worth 10 points, Effective 5 points, Neutral was worth no points, Ineffective -5 points, and Very Ineffective -10 points. The numerical values aren't the critical issue, instead the relative scores between each paradigm are. As you can see, agile and iterative clearly outshine both traditional and ad-hoc approaches, once again I suspect that this is due to the greater discipline of agile/iterative teams (for more on this, see The Discipline of Agile. The greater stakeholder participation, the shortened feedback cycles, the greater focus on quality, and testing earlier in the lifecycle are all factors in the higher levels of success achieved by agile/iterative teams when compared to traditional and ad-hoc teams.
It's interesting to note that although agile and iterative approaches are close, that agile edges out iterative approaches three times out of four. What's more interesting is the differences between the two less-disciplined approaches, traditional and ad-hoc. Although traditional approaches appear to deliver better quality than ad-hoc approaches, having no defined process to follow at all appears to deliver better functionality than traditional approaches (something business analysts should note), appears to be more cost effective than traditional (something that financial people should observe), and is equal when it comes to schedule predictability. Over the past few decades traditional organizations have been making significant investments in the false rigor of bureaucracy and it doesn't appear to have paid off.
Differences Between Project Types
Finally, we wanted to find out if different types of projects had different success rates. Sure enough, they did. Updates to legacy systems had the highest success rate at 78%, followed closely by new development at 76%. Package implementations didn't fare as well at 71% and data warehouse /business intelligence (DW/BI) projects came in last at 69%. I suspect that legacy updates have a higher success rate because they are often smaller efforts done by more senior people. Package implementations and in particular DW/BI projects run into trouble because a traditional approach is often taken to them although as I've written about in the past it is quite possible to take an agile approach to both of them, implying that there's an opportunity to increase their success rates.
All in all, it seems that we're a lot better at software development than we've been lead to believe, although there is clearly room for improvement. I have posted here the original survey questions, the source data from the survey (with identifying information removed), and a PowerPoint slide deck summarizing the results so that you can analyze these results for yourself.
