Defining Success

How do you define project success? Scott provides some suggestions.


October 31, 2007
URL:http://www.drdobbs.com/architecture-and-design/defining-success/202800777

Scott is a DDJ Senior Contributing Editor, Practice Leader Agile Development with IBM Rational, and author of several best-selling books. He can be contacted at www.ambysoft.com/scottAmbler.html.


How do you define project success? Do you consider a project to be successful when it is on time, or is it more important that a system is delivered when it is ready? Is it important to be under budget or should you maximize return on investment (ROI)? In August 2007, I decided to explore how the IT industry defines project success via an online survey (see the sidebar "Survey Respondents" for a description). Although I would have liked more responses from business stakeholders and data professionals, I was still happy with the fact that I received almost 600 responses. As with any survey, the results included a few unsuspected surprises as well as a few important insights into what is actually happening out there in the IT industry.

For years, we've been told that delivering on time and on budget is an important measure of project success. On the surface, this sounds like a good idea, but as you'll soon see it's obvious that these issues aren't as important as they've been made out to be. It would be naïve to expect us to define an industry-accepted definition of project success because the fact is that individual project teams find themselves in unique situations, implying that their definition of success will differ from that of another team. However, it is reasonable to expect to define potential success criteria, and I think that this survey has explored several very important ones.

How We Define Success

By far, the most popular source of IT project success rate statistics is the Chaos Report by the Standish Group (www.standishgroup.com). The third version of this report, published in 2003, notes that 34 percent of IT projects are successful, 51 percent challenged (they are over schedule, and/or they are over time, and/or they are missing significant functionality), and 15 percent of projects are considered failures. These figures never really seemed right to me, even though I'm embarrassed to say that I have referred to them several times over the years, because the Chaos Report was always the generally accepted source. But, if organizations don't actually define project success in the same way that the Standish Group does, then it begs the question what the actual success rates are.

When formulating this survey, I spoke /e-mailed with a number of people as to how they defined project success. I also scoured the literature for various writings on the subject, and as a result of this work I identified five critical factors upon which IT projects are typically judged. These five factors are: schedule, money (resources), scope, quality, and staff. This list isn't complete (for example, a project is sometimes judged on its ability to prove the viability of a technology or technique), but I believe that it does cover the common success criteria.

When it comes to these five success criteria, the survey found:

In short, we appear to value delivering high-quality working software that meets the needs of our stakeholders in a cost effective manner while respecting the needs of the people involved with the project. This is more important to us than building something to specification on schedule and on budget, regardless of the toll it takes on the people involved. Another way to look at it is that people appear to be much more aligned with the philosophies of the Agile community than we are with the rhetoric that we often hear within traditional circles. And we certainly don't like death marches, but then again who really does other than carrion birds?

The survey also asked respondents to rank the five success criteria in order of priority. Overall, respondents indicated that quality was the most important issue, then scope, staff, time, and money respectively. Table 1 compares the rankings of various subsets of the respondents, showing that commercial/private, government, and IT management all indicated the same rankings as the overall group. Stakeholders, nonmanagement IT, and project management all had different rankings, although all categories that I looked at indicated that quality was the most important issue by far.

All/Commercial/Govt./IT Mgmt. Stakeholders NonMgmt. IT Project Mgmt.
1 Quality Quality Quality Quality
2 Scope Scope Staff Scope
3 Staff Time Scope Time
4 Time Money Time Staff
5 Money Staff Money Money

Table 1: Prioritizing project success criteria.

How We're Doing

So, now that we have greater insight into what people actually consider to be IT project success criteria, how are we actually doing in practice? On average, respondents indicated that Agile projects had a 71.5 percent success rate, traditional projects 62.8 percent, data warehouse projects 62.6 percent, and offshoring projects 42.7 percent. All of these rates are a far cry from the 34 percent success rate reported by the Chaos Report.

Figure 1 depicts the success rates for commercial/private organizations versus those of government/public agencies. Although commercial organizations fare slightly better than government agencies, the rates are comparable. Granting that success is being defined in the eye of the beholder, the common belief that the public sector isn't as successful at IT as the private sector may be unfounded.

[Click image to view at full size]

Figure 1: Project success rate by project type and organization type.

Some Worrisome Findings

A substantial proportion of the survey respondents, 105 (17.9 percent), indicated that they were project managers. I was surprised to find that they gave significantly different answers to several questions, particularly when it came to issues surrounding schedule, money (resources), and quality. Table 2 compares their responses with those of other constituents including the average. To be fair only 18 business stakeholders responded—statistically, those figures are on shaky ground (so take them with a grain of salt). It appears that project managers are more interested in delivering on time and on budget than delivering when the system is ready. This may be because of the reward structure in place within the IT organization, although if that was the case, I would expect the numbers to be closer aligned with those of IT managers (and with nonmanagement IT staff for that matter). It may also be the result of different values with the project management community, values that are often driven by the Project Management Institute (PMI)'s Book of Knowledge (www.pmi.org). More research into this divide is definitely warranted.

Project Managers ITManagers NonMgmt. IT Business Stakeholders All
Shipping when the system is ready is more important than shipping on schedule 50.3% 63.1% 64.1% 73.3% 61.3%
Providing the bestROI is more important than delivering underbudget 62.4% 76.9% 71.1% 86.7% 79.6%
Delivering high qualityis more important than delivering on time and on budget 78.2% 87.7% 90.4% 86.7% 87.3%

Table 2: Comparing responses of various constituents.

People's attitude towards a healthy workplace is a serious problem. Although 75.8 percent said that having a healthy workplace is more important than delivering on time and on budget, only 53.3 percent of business stakeholders surveyed said so. I'm happy to say that 80 percent of IT managers and 70.3 percent of project managers put the needs of their staff over being on time and budget, but I'm definitely worried about the business attitudes towards IT staff. This may be indicative of the frustration that business stakeholders often have with IT in general as well as a common belief that IT services are becoming a commodity. Until these figures get closer to 100 percent, I think we have a problem.

I have always believed that if a project team gets into trouble it should either be redirected so as to get it out of trouble or the project should be cancelled. I was shocked to discover that on average only 40.9 percent of respondents felt that canceling a troubled project should be considered successful. Worse yet, only 33.3 percent of business stakeholders felt this way, and only 35.7 percent within the public (government) sector. In my opinion, when a questionable project is started, then may you have portfolio management problems. However, if the project was good at the beginning, but then it ran into trouble, then redirecting or canceling it is a reflection of good portfolio management.

When I give public presentations, one of my favorite questions to ask the audience is whether they've been on an IT project which they knew was going to fail right from the very start. So I asked the same question on the survey, and sure enough the majority of people, 68.6 percent, said that this had in fact happened to them. Sadly, 73.3 percent of business stakeholders, 78.2 percent of project managers, and 76.9 percent of IT managers also indicated this. In other words, the people in a position to get the project on the right track, or at least in a position to influence the people who could do so, couldn't do anything about it. Granted, they may not have been in decision-making positions when they ran into such a situation. Then again, they're now in these positions and are aware of the problem, so hopefully we'll see this problem diminish over time.

Success Is in the Eye of the Beholder

I believe that there are several important lessons to be taken away from this survey. First and foremost, IT project teams enjoy a higher success rate than what we're commonly told. Second, it appears that organizations have their own definition of success, and perhaps even different definitions for different types of projects. Third, Agile software development techniques appear to enjoy a measurably greater success rate than traditional or offshoring projects. Fourth, because we're nowhere near a 100-percent success rate there is still room for improvement. The implication is that software process improvement (SPI) efforts are something that we need to invest in.

I have posted the original survey questions, the source data from the survey (with identifying information removed), and a PowerPoint slide deck summarizing the results at www.ambysoft.com/surveys/ success2007.html. Please take advantage of these resources.

Survey Respondents

The survey was announced on the Dr. Dobb's Report newsletter the last week of August 2007, ran for the entire week, and received 586 responses. As usual, the respondents were fairly senior and they worked for a range of organization size:

  • 73% had 10 or more years of experience in IT
  • 69% were North American
  • 18% were European
  • 84% worked for commercial/private firms
  • 16% worked for government agencies

People's positions were varied, although the majority (53.8 percent) indicated that they were developer/modelers. The number of respondents by position:

  • Business stakeholder: 18
  • NonManagement IT(below)
  •   Data professional: 22
  •   Developer/modeler: 315
  •   Operations/Support: 10
  • QA/Test: 17
  • IT Manager: 68
  • Project Manager: 105
  • Other: 31
—S.W.A.

Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.