Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼

But We're CMM Level 5!

January 2002: But We're CMM Level 5!

Our story takes place at "E-Co," where I was dropped in as the acting Chief Technology Officer. E-Co classifies itself as a "new media company" in the coming media convergence revolution of television and the Internet. It stages media spectaculars, which it then webcasts and uses to market sponsors' products.

Aside from the incomparable quality of our beer, we Canadians love to harp about how our country is the best place in the world to live. For years, according to the United Nations Human Development Index, which measures quality of life by factors such as education and life expectancy as well as per capita income, Canada ranked number one. This gave many Canadians license to chide our American friends about their country's poor showing (merely number 12). UN officials repeatedly scolded Canadians because the index was never intended to be misused in this manner—it was designed to classify countries for development programs, not for propaganda purposes. (By the way, you don't hear a lot about the index around here anymore after Canada slipped to third place this year behind Norway and Australia.)

Just as countries can sometimes misuse classification models to lay claim to a superior quality of life, the software industry misuses classification models like the CMM to claim superior product quality.

Classifying Maturity
The Software Engineering Institute's (SEI) Capability Maturity Model (CMM) is a five-step model for ascertaining the maturity of software development organizations. This score is determined with a survey of the company's compliance with the CMM's Key Practice Areas (KPAs). KPAs are what I would normally think of as industry best practices, including version control and defect tracking. The majority of software development organizations are level 1, which is considered random or undisciplined processes. Companies that follow minimal best practices such as defect tracking and version control move up a step to level 2. Institutionalize these practices, and you climb another step up the maturity ladder.

The CMM was originally developed to provide the government an objective scale to evaluate its contractors' software development maturity. It has become the means by which many in the software engineering field rate the maturity of their software development organization. It's a good model for evaluating your software engineering capabilities and identifying areas that you can improve on, but, just like the UN Human Development Index, it can be misused for political and commercial purposes.

Case in point: A while ago, I was talking to the development manager of a company that creates human resources management software, discussing the bids that I had received from different vendors for our e-commerce system, some of them in the million-plus region. He stopped short of telling me that I was insane to outsource to North American software houses, but enthused about his company's experience in outsourcing to India. For just 40 bucks an hour, he said, I'd have access to top-notch programmers.

I remembered that, in university, a lot of my friends were visa students from India. Many stayed in Canada, but quite a few returned home. They were good developers, and if I could hire high-quality talent for $40 an hour instead of the $175 an hour I was being quoted by most development houses—well, my friend was right: I'd be crazy to outsource to a North American software house.

My colleague introduced me to their outsource vendor's agent, a pleasant, enthusiastic chap who gushed about his company's technological prowess: leading-edge software technology, object oriented, RUP, Use Cases, UML, Rational Rose. And, about every fourth sentence, he was careful to mention that they were CMM level 5: To maintain their certification each year, they had to fill out the CMM survey and identify their implementation of the CMM KPAs to the auditor's satisfaction.

Though I view certification as a good sign, I'm always a little skeptical of a company that considers certification a guarantee of quality. A few years ago, I participated in a local corporate consortium about ISO 9000, another certification for process quality. Our favorite joke was "You could make lead life jackets and still be ISO 9000 certified." Also, I'm a little cautious about certification processes in general—friends working in the defense industry have told me too many stories of being "coached" before filling out various quality audits.

My friend showed me a sample of the company's work products, starting with use cases. They were all CRUD-type use cases, and I could classify them as either Create X, Read X, Update X or Delete X—use cases in name only. Though they may have satisfied the CMM requirements, they certainly didn't delineate the value of the system to its users. The class model was a poor combination of massive "God" classes and data classes, with nothing more than set/get operations for every attribute. There was nothing object oriented about the architecture—a CASE tool like Rational Rose isn't a magic OO wand, and automatic code generation from a bad model does not create great code. The use of such tools may be a sign of software maturity, but it's no guarantee of product quality.

Caveat Emptor
Despite my concerns, my friend was happy with the vendor because he was receiving the benefits of a mature software development process with regular, predictable delivery of product at a defined level of quality—at a bargain cost. Unfortunately, sometimes you get what you pay for. With minimal software experience, my friend wasn't aware of the product's brittle architecture, nor did he anticipate the high cost of inevitable feature changes and enhancements in the future. As his product was designed to serve as the framework for a family of related products, this was a real disaster in the making.

With E-Co, I'd flirted with disaster long enough. When I told the vendor agent that we wouldn't pursue business with them, his entire counterargument was, "But we're CMM level 5!"

Maturity Mantra
Well and good—but I wanted more. Karl Wiegers wrote: "The CMM assumes that God gave you the requirements and you take them from there." My requirements specifications were anything but divinely inspired. I needed a vendor who would work with me to quickly resolve requirements issues, with the ability to recommend and implement improvements. The vendor's mantra "We're CMM level 5!" made me wonder if that certification would be used as a shield to deflect any complaints, framing product defects as E-Co's responsibility because of our imperfect requirements. I simply wasn't confident that this vendor had the ability—or desire—to protect E-Co's interests.

While I strongly believe in a disciplined software development process, a high CMM level does not necessarily imply high-quality products. Great configuration management means that you'll always know what went into a build, but it won't improve the flexibility of the software in that build. Great defect tracking means that you probably won't ship software with the same defect twice, but it won't guarantee a great solution for that defect. To produce high-quality software, you still need a high-quality software development team.

Moral: There are lies, damn lies and maturity models. Beware of attempts to use the CMM beyond a statement of software development process maturity.

Next: First Flight

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.