Channels ▼


The Non-Existent Software Crisis: Debunking the Chaos Report

The software development success rate published in the Standish Group's Chaos Report is among the most commonly cited research in the IT industry. Since 1995, the Standish Group has reported rather abysmal statistics — from a rate of roughly one-in-six projects succeeding in 1995 to roughly one-in-three projects today. Ever since the Chaos Report was published, various Chicken Littles have run around warning about how this "software crisis" will lead to imminent disaster. However, this supposed "software crisis" is complete and utter hogwash, it always has been and I suspect always will be. Sadly, that doesn't stop people who should know better, or at least should be able to think for themselves, to continue quoting this nonsense.

Since 2006, I have organized an almost-annual survey for Dr. Dobb's that explores the actual success rates of software development projects. The most recent was conducted in November and December of 2013 and had 173 respondents. The original questions as asked, the source data, and my analysis can be downloaded for free at 2013 IT Project Success Rates Survey results. The survey was announced on the Dr. Dobb's site, on the Ambysoft announcements list, my Twitter feed, and several LinkedIn discussion forums. The results of this study are much more positive than what the Standish Group claims. They still leave significant room for improvement, but they certainly don't point to a crisis.

The success rates by development paradigm are summarized in Table 1. As you can see, all paradigms (even an Ad hoc approach) fare much better than a one-in-three success rate. In this study, we asked people to judge success based on criteria that was actually applicable to the team at the time. In this case, a project is considered:

  • Successful if a solution has been delivered and it met its success criteria within a range acceptable to the organization;
  • Challenged if a solution was delivered, but the team did not fully meet all of the project's success criteria within acceptable ranges (for example, the quality was fine, the project was pretty much on time, but ROI was too low);
  • Failed if the team did not deliver a solution.

















Ad hoc








Table 1: Software development success rates by paradigm.

Each paradigm was well-defined. Respondents were first asked if their organization had any project teams following a given paradigm, and then what percentage of projects where successful, challenged, or failed. A weighted average for each of level of success was calculated. A selection of 91-100% was considered to be 95%, 81-90% as 85%, and so on. A selection of 0 was considered as 0, and answers of "Don't Know" were not counted. I then had to normalize the value because the weighted averages didn't always add up to 100%. For example, the weighted averages may have been 60%, 30%, and 20% for a total of 110%. To normalize the values, I divided by the total, to report 55% (60/110), 27% (30/110), and 18% (20/110).

Defining the Paradigms

The paradigms in this survey were defined as follows:

  • Ad hoc. On an Ad hoc software development project, the team does not follow a defined process.
  • Iterative. On an iterative software development project, the team follows a process that is organized into periods often referred to as iterations or time boxes. On any given day of the project, team members may be gathering requirements, doing design, writing code, testing, and so on. An example of an iterative process is RUP. Agile projects, which are defined as iterative projects that are performed in a highly collaborative and lightweight manner, are addressed later.
  • Agile. On an Agile software development project, the team follows an iterative process that is also lightweight, highly collaborative, self-organizing, and quality focused. Examples of Agile processes include Scrum, XP, and Disciplined Agile Delivery (DAD).
  • Traditional. On a traditional software development project, the team follows a staged process where the requirements are first identified, then the architecture/design is defined, then the coding occurs, then testing, then deployment. Traditional processes are often referred to as "waterfall" or simply "serial" processes.
  • Lean. Lean is a label applied to a customer-value-focused mindset/philosophy. A Lean process continuously strives to optimize value to the end customer, while minimizing waste (which may be measured in terms of time, quality, and cost). Ultimately, the Lean journey is the development of a learning organization. Examples of Lean methods/processes include Kanban and Scrumban.

Ad hoc development (no defined process) and the Traditional approach had statistically the same levels of success in practice. Our previous studies have also shown this result. However, when you take team size into account, Ad hoc is superior to Traditional strategies for small teams, yet the opposite is true for large teams. Agile and Iterative strategies had similar results on average, which we've also found to be true in the past, regardless of team size. For the first time, Lean strategies for software development, supported by both Kanban and the DAD framework, were notably more successful than the other four paradigms we explored. Food for thought.

Time to Get Lean?

For each paradigm, we also looked at several potential success factors. The questions we asked were:

  • Time/Schedule. When it comes to time/schedule, what is your experience regarding the effectiveness of [paradigm] software development teams?
  • Budget/ROI. When it comes to effective use of return on investment (ROI), what is your experience regarding the effectiveness of [paradigm] software development teams?
  • Stakeholder Value. When it comes to ability to deliver a solution that meets the actual needs of its stakeholders, what is your experience regarding the effectiveness of [paradigm] software development teams?
  • Product Quality. When it comes to the quality of the system delivered, what is your experience regarding the effectiveness of [paradigm] software development teams?

For each success factor, respondents were given options of Very Effective, Effective, Neutral, Ineffective, Very Ineffective, and Not Applicable. For each question, we calculated a weighted average by multiplying the answers by 10, 5, 0, -5, and 10 respectively (answers of "Not Applicable" were ignored). By doing so, we're now able to compare the relative effectiveness of each paradigm by success factor, which you can see in Table 2. It's interesting to note that Lean approaches were perceived to provide better results on average than the other paradigms. Also, the three development paradigms of Lean, Agile, and Iterative were significantly better across all four success factors than either Ad hoc or Traditional (waterfall). In the vast majority of cases, only cultural inertia can justify a Traditional approach to software development these days.




Stakeholder Value

Product Quality
















Ad hoc










Table 2: The effectiveness of each software development paradigm.

How Is Success Defined?

The survey also explored how people define success, which I argue is exactly where the Chaos Report falls apart with respect to measuring project success rates.  Similar to what I found in the 2010 study, there was a robust range of opinions when it came to defining success:

  • Time/schedule: 16% prefer to deliver on time according to the schedule, 39% prefer to deliver when the system is ready to be shipped, and 42% say both are equally important
  • Budget/ROI: 13% prefer to deliver within budget, 60% prefer to provide good return on investment (ROI), and 23% say both are equally important
  • Stakeholder value: 4% prefer to build the system to specification, 86% prefer to meet the actual needs of stakeholders, and 10% say both are equally important
  • Product quality: 10% prefer to deliver on time and on budget; 56% prefer to deliver high-quality, easy-to-maintain systems; and 34% say both are equally important

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.



I'm joining the conversation a year later, but I really appreciate the topic as I model business processes and write software requirements for a living. I am acutely aware of what are the factors for success on a software project. The Standish group does pretty good work collecting and selling this data. 'Crisis' is likely a reasonable term to use when they first started to publish the data and conclusions. I agree that the definition of success varies and its not fair to judge a project for success when there are so many variables including the original estimates which usually miss the mark.

What I did not see in Mr Amblers survey questions or conclusions is the dollar or currency cost of the project as a determining factor of success. I suspect that the high rate of success reported is for projects budgeted/cost for $1-3 million or less. I have not seen the Standish group data recently, but the 2011 success rate for a project costing $1 million or less was 73%. Over $10 million was 7% successful and 74% challenged.

This make sense to me since a larger project dollar value cost is longer term, contains more unknown variables, and are more difficult to estimate. A smaller project is easier to see the end of, variables are better known.


The PROBABILITY OF A PROJECT, P, being completed on time and within budget contraints may be expressed as the ratio:
P = 1/B*u, (B raised to the power of u), where B = Budget and u = urgency.


Hi Scott

This is comment is a year late. However, I just came across this conversation, and thought that I would add my 10 cents worth.

My initial thoughts on the Chaos report was:

- Yes, there is a significant problem with Software projects. This is my gut feel talking, after running and reviewing IT projects for more than 17 years.
- However, the numbers being reported may not be right. Obviously, I have no basis to question the data.

The potential root causes of the Cost overrun problem has been studied by Jorgensen M & Molokken K in their paper "How large are the Software Cost overruns? A review of the 1994 Chaos report", as commented by "Malicorneous" in his comment on your Blog.

But I believe that a few root causes may have been missed which contributed to this very large skew of results:

1. There is no "acceptable range" defined for success. Very few projects would complete with 0% variance on cost or schedule, but Organizations may have a defined "Cost variance range" that is acceptable as "success" (Eg, 3% Cost variance). So having a 2.5% Cost variance would be considered as successful by the Organization, but this project would be treated as a "Challenged" project by Standish group. Similarly, a 5% variance on schedule may be within the acceptable limit for the Customer, and may well have been contracted in this manner.

2. The acceptable range of variance would differ for various types of projects, specially in terms of quality. A Life-monitoring system used in an ICU or an Aircraft navigation system may need reliability that is even greater than 6-Sigma and is as close to 100% as possible, but a system that monitors Air conditioning quality & gives directions to the cooling plant would require a much lower level of reliability. In case of the former project, the level of reliability required would actually be a feature of the project, and would need to be factored into the project as such, with appropriate architecture, infrastructure, and software design being applied. Comparing these two types of projects while ignoring the "acceptable ranges" would, I believe, lead to misleading results. So bottomline: There would be a different set of acceptable ranges for different types of projects. We need to give this flexibility to the Project teams when reporting their success and failure rates.

3. The appropriate set of respondents who should have been invited to provide their inputs, at least on failed software projects (as well as highly successful ones) should be the Customers who funded the projects, not the IT Organizations themselves. The Customers would have been able to provide a facts-based response specially in case of failed projects, whereas this picture may not have been available from the IT organizations, given the voluntary nature of survey responses. After all, no one wants to paint themselves black, even if the inputs are being fed into a survey tool with no attribution - a big "what if" thinking would probably prevent this kind of disclosure.

Another perspective on the study that you have conducted is:
1. Organizations undertaking Lean & Iterative projects are typically higher on the Project Management maturity scale than the ones executing Traditional & Ad hoc projects. So the reason for a higher percentage of failed projects in traditional & ad hoc projects is clearly apparent.
2. The values reported against Adhoc projects look better than Traditional projects. I believe that the root cause for this anomaly is that organizations would execute very small projects in the adhoc style, whereas larger (medium-size to transformation projects) would be executed in Waterfall model or another mature SDLC model.

Your thoughts?


No software crisis huh? The education system in South Africa is one of the lowest in the world. In order to cover up the education crisis, the government drops the pass rates. Now every one is passing, and no one in this eco system, the student, the teacher, the parent will tell you any different than a success story. This is human nature. In 1998 Thabo Mbeki handled the AIDS crisis, by simply saying that HIV did not cause AIDS. Hmmm wonder how that worked for them? Again this is human nature. Our strongest emotion is fear, and it is easier to deny a problem exists than to face it.

The average developer today, cannot even deal with the concept of a simple thread. I know I have interviewed more than 100 so called developers, that work on "successful" projects, and they are simply clueless in the most basic constructs of computer science. (This is normal today ... there is google)

But all this is forgiven in our industry as we lean on the proverbial agile crutch. We run an iteration, mess it up get feedback and, re re re re factor and more re re re factor. We call this progress and accept it as normal healthy agile. Developers copy and paste, code from google searchers trying to get it to "WORK" so they can deliver at the demo meeting. Nine out of ten, times a developer cannot even explain the code in there application, as they copied and pasted it from somewhere else ... justified by ... it works !!! ...

Call it what ever you like. My brain is not in neutral, when I am sick I go to the doctor. I do not Google it, or try some experimental drug and come back in 2 weeks to try something else. Sounds crazy? Apparently this is the definition of healthy Agile, Don't laugh this is our industry. We take on very expensive projects and with out plans and start laying bricks, only to knock it down next week. I wonder how much code is thrown away everyday ... But no ... No crisis.

I run an Agile shop, and am highly successful using volatility based decomposition. I do not build my systems based on requirements, but rather capture the essence of the problem domain, in a service orientated architecture. I design for the only constant which is change. I don't write code against user requirements. I rather orchestrate the required run-time behavior as the business developments of the day calls for it.

Yes this is harder, and requires more skill, but you simply cannot avoid the hard work up front, to get out anything of value. This is why we do not have perpetual energy sources and why alchemy fails.

I see every project that requires a massive cost of ownership, in terms of maintenance, flawed in security, and failing under load as a failure. By definition, that leaves a lot of broken junk in the world or at least a lot of applications that simply cannot scale out to the changing requirements of the business. And yes I am very sure they worked yesterday.

But do not worry. The very people who bring you this, says, that everything is under control. No crisis. We are AGILE.

Just because you have never seen a successful project, and a successful implementation does not mean it cannot be done, all it means is that you have never seen it before.


" we'll find out about this system as we go."

I pray to god you're never involved in any software that determines my life.


Software needs engineering the very same reason bridges need engineering. Just because it's hard doesn't mean we give up on engineering!

When software projects collapse they can just as directly lead to people dieing as well as a bridge collapsing. "well we're sorry the laparoscopic camera had a software bug and we perferated your colon, you'll just need to wear a colostomy bag".


"What it measures is whether project teams are reasonably on time, on budget, and are building something to specification."

This is the definition of success.

Anything other than this is..... not success. You can pick your words to describe this state, but there is absolutely one word you cannot use: success.

Success is delivering a project that delivers business value meeting the stakeholder's needs on time and on budget.

Anything else is not.

"How many organizations have you seen with a less than one third success rate? Any? I get around, at one point in my career I was visiting 20+ organizations a year, and I've never run into one with such an abysmal track record."

Because they lie. They lie to themselves. They don't want to admit they fail 66% of the time, so they move the goal posts so they can sleep better at night. They move their goal posts so the shareholders don't riot and toss out the board of executives at every shareholders meeting.


Hogwash indeed! Sorry Scott – if the organization chooses to measure success as something other than “on time, on budget, with expected functionality” – then one can easily see how there is no ‘crisis’. Such organizations have opted to accept something less than what they ought to be striving for.

Analogy – President Kennedy challenges the USA to put a man on the moon, and return him safely to earth, by the end of the ‘60’s. Drop the “return him safely” from the mission and then call getting the astronaut there but having him perish because we couldn’t get him back a success! Well, it meets the criteria, but is it really a ‘success”?

Another (real life) analogy – my wife asks me to clean the bathroom. By whose criteria should we measure success? If it is mine, then this will take about 10 minutes. If it is my wife’s criteria, there may be a second (and maybe third) iteration involved! Should we really accept the fox as the keeper of the hen house?


I agree with Tlingit. Ambler allows his respondents to essentially redefine success as "Time/Cost/Functionality - pick two out of three." What other industry allows success to be defined this way?


Yep, that's the problem. ;-)

Seriously though, it's a lot of work to do deep research as you suggest. This work is difficult to fund and even more difficult to get support for within target organizations to study.

I often ask my clients if they would be willing to share their metrics publicly, yet few are willing to do so. This is a fairly easy thing for companies to do, particularly compared to starting up a new study with a researcher, but they are always worried about losing some sort of competitive advantage by doing so.


I agree entirely. The "Software Crisis" has been touted as critical for much of the time I've been in software (which would be 41 years). The Standish Group's Chaos Report is routinely quoted as a source of this and a measure of the depth of the crisis. But how do they collect data?

Molokken and Jorgensen gave an excellent critique of the Standish Group's collection and analysis process (whatever that might be) as well as their conclusions [see]

Having done this a lot over the years, collection of reasonably indicative data is very difficult. Projects which run badly rarely collect data on how badly they run. Projects which utterly "fail" do not, as a rule, stand up to be counted. Quite the opposite: usually they bury the dead bodies and pretend it never happened. So there is certainly some survivor bias in the numbers. But the real criterion for "success" is the delivery of value over cost, not whether a project made a particular target budget or date.

It is clear to anyone who has engaged in any large-scale systems development that being agile is much better than not being agile. It's also clear that the early commitment of large amounts of money when little is known about the system being built or what might be encountered while building it is really risky. This was true back in the sad bad days of waterfall projects and it's true today.

The best rejoinder to the criticism that software costs too much came from Tom DeMarco (in, well, "Why Does Software Cost So Much?") when he observed: "...compared to what?..." If the Software Crisis is really so bad and has continued so long, wouldn't the marketplace have replaced it with something more efficient?


Isn't this a bit of a rehash of Robert L Glass's 2005 article in the IEEE Software magazine?

The Time, Scope, Budget measures of success are the 'Gold Standard' for project management and are, unfortunately, still held in high-esteem by many project managers.

There are literally dozens or research papers on the subject of software project success/failure and the need to have them agreed upfront. This is, of course, the basis of Tom Gilb's Evo method.

Collecting anecdotal evidence from arbitrary participants seems kind of random, no matter how you crunch the numbers. As many others have pointed out success is in the eye of the beholder.

Like Scott, I visit many different organisations every year. There only two reasons they call on me. Either they have (real or perceived) problems with delivery or they just want some training. Usually it's the former.

You can bet your bottom dollar that if the Project Managers call me in and tell me they haven't delivered a project successfully (if at all) in the last three years, the Engineers will tell me every project they do is successful.

Conversely, if the Engineers call me in and tell me there are project problems, the Project Managers will deny vehemently it.

It may be the case that I only see the problem organisations because that's the sort of work I do but my experience tells me there are lots of problems out there.

Maybe we're suffering from too much of this:


If you think that there is no software crisis, explain why the NASA contractor tried to do a 1B U$ logistics software and nothing came out of it, or tell me why in IT help desks, there are no responses to basic problems, there no standard practices, tell me why hardware logic verification is so complicated and needs very smart people, and not kids trying to have fun making some script to test a program, tell me ways to know what happened in the unintended acceleration case against Toyota in the US inboard computer and nobody even NASA knows a **** about how it could happen......... If you can explain me all of that then your column is right.


I wish! In fact, I was kind of hoping you and/or would want to take that on. :-)


Hi Scott

I've long suspected the Standish report and its claims, as the survey methodology is opaque, and of course, the hype is self-serving.

As far as I can tell, the participants in your studies are self-selected and you do not attempt or claim broad representativeness. So these results cannot be viewed as anything but a collection of profile case studies from persons who are inclined to tell their story. Typically, people are not inclined to self-report failures, so it seems likely to me that projects that failed (by any reasonable definition) are probably under-represented. Many, if not most, enterprise IT and ISV organizations stand on a large legacy codebase, typically the result of thousands of person-years of development using what you call "traditional paradigm." That would count as a success in your terms, but I suspect little of that is reflected in your survey responses.

I don't say this to diminish your results - they are interesting and instructive in their own right. I've done several such studies and have concluded there is no cheap way to achieve reliable representativeness, but it is important not to read too much into these snapshots.

NIST did a plausible macro-economic analysis about ten years ago, and concluded at that known-bad IT practices add about $50B of avoidable cost to the US economy annually. There are many other credible studies. I don't know of any that have found evidence of a "crisis."

Of course, there are plenty of horror stories. Here's a case study of how bad software development lead to the collapse of a successful ISV

All this indirectly raises the question of success. I like Tom DeMarco's reflection on all of this: "Consistency and predictability are still desirable, but they haven’t ever been the most important things. For the past 40 years, for example, we've tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business." IEEE Software, July/August 2009.


I agree with what you're saying. Can you point to a study that has done those sorts of things that you write about?


There are some things that are very useful from engineering that can be used in software development. I agree. But there are many things that are different about software development for which engineering is not suited. Maybe I can recommend reading some writings of David Parnas, one of the most disciplined of software developers ever.

Engineering is much more planned - you have to get it right before it goes into production. Software is by nature much more experimental - we'll find out about this system as we go.


Well, I suggest you read Koch and Godden's book. They are management consultants. But actually, the function of management has not disappeared, it is just the need of a separate class to have to do it.


Thanks for responding, Scott. I guess I already did check my gut because, like I wrote before, "I don’t know that I totally believe the Chaos Report."

To me, if someone wanted to do a useful study of software success and failure, they would need to select specific projects that represented the diversity of projects that are being built... varying scopes, technologies, industries, countries, etc. Then they would need to survey three groups: those who built them, those who use them, and those who paid for them. Only then could they compare findings to try to understand what has been happening out there.

I bet Microsoft felt pretty good about Windows 8.0, and Apple felt good about iWork 2014. It wasn't until those products hit the street that they got their reality check.

Consequently, if someone is trying to come up with software industry success statistics by surveying software people -- whether it is Standish Group, you or the NSA -- then their approach is fundamentally flawed.

Software people declaring their software projects to be successful is like men declaring themselves to be great lovers. I'm glad they feel like they're getting positive feedback and all, but they aren't the group that actually knows.

Success isn't meeting a checklist of criteria, success is content, repeat customers.

I'm currently finishing a software project that was supposed to be finished February 2. It will be actually finished on February 7. It is slightly under budget and the client loves it so much they've been demoing the work in progress to their clients these last 2 weeks.

Was this a successful project then?

I say we are deceiving ourselves if we think we know that today simply because we met certain criteria.

Everyone thinks their babies are smart, beautiful and perfect when they are born... it's when you are sitting in the principal's office for the second time in a week that you start to get real clarity on the job you think you've done.


"Whether or not you like the article, one thing is true. We need to get away from the notion of 'software crisis'."


Maybe it's me but I see no correlation between how hard something is to change, and the use of the word "engineering."

Of course software should be engineered -- any complex outcome should be engineered, whether it is computer hardware, computer software, a movie screenplay, or date night. Flexibility is helpful, but its not a replacement for engineering.


Actually, the definition of challenged reflects what the Chaos Report uses for challenged.

The fundamental difference between the Chaos Report and my study is that the Chaos Report forces a definition of success on teams that is rarely applicable. My study levels the playing field by measuring success based on the criteria that was actually applicable for the given team.

Do a quick gut check though. How many organizations do you know with a one third success rate at software development projects? If one third is roughly correct, then roughly half of the organizations that you're familiar with should have less than a one third success rate and half would have more than that. So, is this a fair assessment of what you're able to actually observe yourself? Or, is a roughly two-thirds success rate a bit more realistic (based on your own observations)?


You had me until you got to "management is just a relic of the industrial revolution."

Actually, management seriously predates the industrial revolution, and is still necessary today --

Just not the way it is usually done today.

Have you heard of "radical transparency?" I know a software company that's building great stuff with an extremely flat (and minimal) management structure.

Check out this New York Times interview of Jeff Smith (of Qualtrics). I know Jeff, and his mom is our good friend and neighbor... impressive stuff.


Whether or not you like the article, one thing is true. We need to get away from the notion of 'software crisis'. This comes from the mistaken belief that software development should be like hardware engineering (hence the term software engineering should be banned as meaningless). Hardware is difficult to change - so we have software which is far more flexible - that is the cause of the problem and the 'crisis'. But it is the very nature of software to be like that. Use that nature for its strength, not its weakness.


I'd say, no the developers are not out of touch - they are the ones in touch with the work. They should be a part of requirements gathering, but too often are kept away from that process by managers trying to maintain a power base - so they could be out of touch in that regard. But that is not their fault.

Managers who can't do the development work are the ones out of touch and this scares them - thus a whole lot of tactics are employed in the work place to maintain this power base.

But management is just a relic of the industrial revolution. Managing should be a part of every person's job, not a separate activity.