The results of the process framework survey were a surprise

March 26, 2008
URL:http://www.drdobbs.com/tools/dr-dobbs-agile-newsletter-0308/206905819

In This Issue

• How Effective are Process Frameworks?
• Hot Links

How Effective Are Process Frameworks?

In January, 2008, Dr. Dobb's ran a survey which explored the adoption rate and effectiveness of various process frameworks, in particular COBIT, CMMI, ITIL, PMI PMBoK, Prince2, TOGAF, and Zachman Framework. The survey ran for a week and it was promoted in Jon Erickson's blog and in a mailing which went out to Dr. Dobb's readers. The response rate was unusual for us, I suspect because of the topic: Only 339 started the survey, the lowest survey response rate that we've ever gotten, and only 219 completed it -- many people were turned off by the second of five pages which explored the adoption rate and effectiveness of frameworks listed above. Anyway, the results are interesting and arguably contradict some of the marketing rhetoric that we've heard over the years concerning some of these frameworks.

Let's start with a brief overview of each framework. This is important because as I discuss below the survey showed that many people are not familiar with these frameworks, even though they very likely should be. The frameworks I considered were:

• Control Objectives for Information and related Technology (COBIT). A framework for information technology (IT) management which provides managers, auditors, and IT users with a set of potential measures, indicators, processes and practices to assist them in maximizing the benefits derived through the use of IT and developing appropriate IT governance and control in an organization.

• Capability Maturity Model Integrated (CMMI). A requirement specification for a development process promoted by the Software Engineering Institute (SEI).

• The Information Technology Infrastructure Library (ITIL) is a set of concepts and techniques for managing IT infrastructure, development, and operations developed by the UK's Office of Government Commerce (OGC). ITIL gives a detailed description of a number of important IT practices with comprehensive checklists, tasks and procedures.

• Project Management Institute (PMI) Project Management Book of Knowledge (PMBoK). The PMI publishes standards related to project management and manages project management certification, its main standard is the PMBoK which is the only ANSI standard for project management.

• Projects in Controlled Environments v2 (Prince2). A project management methodology which covers the management, control and organisation of a project. Prince2 is promoted by the U.K.'s OGC.

• The Open Group Architecture Framework (TOGAF). An enterprise architecture framework which provides a comprehensive approach to the design, planning, implementation, and governance of an enterprise information architecture.

• The Zachman Framework. An enterprise architecture framework which describes a taxonomy for organizing architectural artifacts, such as design models and requirements documents. The taxonomoy takes into account both who the artifact targets and what particular issue is being addressed.

For each of these frameworks we asked whether your organization had adopted it and if so then how well it was working. For each framework respondents were asked "How would you describe your organization's adoption of [FRAMEWORK]?" and asked to choose one of Very Effective, Effective, Neutral, Ineffective, Very Ineffective, Just started the program and it's too early to tell, We considered it but decided not to adopt, We considered it and adopted some ideas from it, and I've never heard of this framework.

To determine how effective the frameworks are in practice, I took a weighted average of the responses by assigning the following points to each answer: Very Effective (2), Effective (1), Ineffective (-1), Very Ineffective (-2), and adopting some ideas (0.5). All other responses earned zero points. I also took a look at the responses from two groups so that I could compare the results to try to identify the true range of potential effectiveness of these frameworks. The first group was senior staff (100 respondents) which I defined as architects, IT managers, process engineers, and portfolio/program managers and the second group was developers (106 respondents). The results, in the format FRAMEWORK (Senior Staff Rating/Developer Rating), are: COBIT (0.245/0.019), CMMI (0.065/0.014), ITIL (0.15/0.019), PMI (0.18/0.009), PRINCE2 (0.075/-0.014), TOGAF (0.09/-0.009), and Zachman Framework (0.11/-0.005). As you can see, the developers weren't as impressed with the frameworks as senior staff were, I suspect because developers naturally chaff at any controls or constraints put on them. Even so, even senior staff weren't very impressed with the effectiveness of these frameworks " not one framework was even close to be considered effective (a score of 1) on average. Arguably these frameworks aren't living up to their promises in practice.

I was also a bit shocked as to the lack of market recognition that many of these frameworks have, even though all have been around for many years. The Prince2 numbers were understandably low because the majority of respondents, 61 percent, were from North America where the PMBoK overshadows Prince2. However, that doesn't explain the low recognition rate of the other frameworks. For example, IT managers should have at least heard about both COBIT and ITIL, yet respectively 52 percent and 32 percent hadn't. Similarly, architects should have heard about TOGAF and the Zachman Framework, yet respectively 54 percent and 38 percent hadn't. Everyone should have heard about CMMI, yet 23 percent hadn't. The implications are that the groups behind these frameworks still have significant outreach to do, and that any claims by some of the promoters behind these frameworks that they're offering "industry standards" are questionable at best.

All of these frameworks definitely have value when implemented appropriately within context, but the mixed adoption results appear to show that many organizations are struggling to do so. Implementing process frameworks straight "out of the box" likely isn't going to work very well for you, but picking practices from each one and developing an IT process which makes sense for your situation can work well if you have the expertise to do so. Getting help from people who have been there before is likely the best advice that I can give you.

Hot Links

The results of the survey, including the original questions as they were asked and the source data, are available here.

Extending the RUP with the Zachman Framework describes how to tailor the framework into the Unified process.

My Agile@Scale blog is here.

Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.