Since 2007, I have actively explored the success of IT projects in practice via our annual Dr. Dobb's IT Project Success survey. In every iteration, we discover some interesting tidbits of information, and this year was no exception. The survey was comprehensive, with 50 questions in all. It explored the overall success rates of project teams following five paradigms (agile, iterative, lean, traditional, and ad hoc), their ability to deliver on various success criteria (quality, business value, return on investment, and schedule), and the respondents preferences for how they like to address those success criteria.
We ran the 2011 survey during the last two weeks of October and we obtained responses from 178 respondents. As usual, we had a robust range of respondents: 27% were developers or modelers; 51% were managers or team lead/Scrum masters; 84% had 10 or more years in IT; 25% worked in organizations of 500+ IT people; 51% were North American, 25% European, and 18% from Asia Pacific countries. The survey was announced in October on Dr. Dobb's, and at my survey page, via Twitter, through my announcements list, and via several LinkedIn discussion forums. The data, summary, and slides are downloadable from my survey page free of charge.
Defining the Development Paradigms
Over the years, I've kept the descriptions of the various development paradigms stable to aid in year-on-year comparison. Of particular note, this was the first year we explored lean on its own: To develop the definition, I worked with two well respected published authors in the lean software space.
The paradigm definitions used in the survey were:
- On an ad hoc software development project, the team does not follow a defined process.
- On an iterative software development project, the team follows a process that is organized into periods often referred to as iterations or time boxes. On any given day of the project, team members may be gathering requirements, doing design, writing code, testing, and so on. An example of an iterative process is RUP.
- On an agile software development project, the team follows an iterative process that is lightweight, highly collaborative, self organizing, and quality focused. Example of an agile process is OpenUP, Scrum, or XP.
- On a traditional software development project, the team follows a staged process where the requirements are first identified, then the architecture/design is defined, then the coding occurs, then testing, then deployment. Traditional processes are often referred to as "waterfall" or simply "serial" processes.
- Lean is a label applied to a customer value-focused mindset/philosophy. A lean process continuously strives to optimize value to the end customer, while minimizing waste that may be measured in terms of time, quality, and cost. Ultimately, the lean journey is the development of a learning organization. Examples of lean methods/processes include Kanban and Scrumban.
Which Development Paradigm Works Best?
The primary goal of the survey was to determine the average success rates, by paradigm, of IT development projects. Because individual project teams have different success criteria, I asked respondents to indicate the level of success of their project teams based on the actual criteria faced by the teams. Other studies, in particular the Standish Group's Chaos Report, impose a definition of success, and then measure against that. A commonly imposed definition is whether the project team was relatively on time, relatively on budget, and reasonably implemented to specification. Imposing any definition of success, particularly that one, is clearly inappropriate as I'll soon show.
In the case of this survey, a project was considered successful if a solution was delivered and it met its success criteria within a range acceptable to the organization. A project was considered challenged if a solution was delivered, but the team did not fully meet all the project's success criteria within acceptable ranges (e.g., the quality was fine, the project was pretty much on time, but ROI was too low). A project was considered a failure if the project team did not deliver a solution at all. In future surveys, I think I will rework the failed definition to exclude projects that were purposefully cancelled with the full agreement of the stakeholders.
With these definitions, I found that iterative and agile approaches had statistically the same success rates, with 69% of iterative projects being successful and 25% challenged; whereas 67% of agile projects were considered successful and 27% challenged. Previous surveys also found that agile and iterative had statistically similar success rates. Interestingly, the success rates have inched up a bit from last year, although not in a statistically significant manner.
Traditional and ad hoc teams also statistically had the same levels of success, which in turn, were statistically lower than those enjoyed by agile or iterative teams. In this case, traditional teams were 50% successful and 36% challenged, and ad hoc teams were 49% successful and 38% challenged. This is down a bit from last year, although once again, not in a statistically significant way. So once again, on average, it appears that either agile or iterative approaches to software development prove to be less risky in practice than traditional approaches.
In the past, I've argued that the primary cause for the difference in success rates is that agile and iterative approaches require greater levels of discipline than do traditional or ad hoc approaches. This opinion has caused a bit of controversy with traditionalists, as you could imagine, as they believe they are working in a disciplined manner as well my belief is that there is a serious difference between bureaucracy and discipline, making me question the "discipline" of traditional software development.
This survey was the first time we asked about lean. We only had 40 responses, so the results are statistically suspect this year and I hope for better in the future. Ignoring statistics for a moment, the survey found that 62% of lean projects were considered successful and 30% challenged, putting it dead center between the agile/iterative and traditional/ad hoc pairings. This may be a reflection of the fact that people are applying lean techniques in a range of situations, including all four of the other paradigms.