Our story takes place at "E-Co," where I was dropped in as the acting Chief Technology Officer. E-Co classifies itself as a "new media company" in the coming media convergence revolution of television and the Internet. It stages media spectaculars, which it then webcasts and uses to market sponsors' products.
Aside from the incomparable quality of our beer, we Canadians love to harp about how our country is the best place in the world to live. For years, according to the United Nations Human Development Index, which measures quality of life by factors such as education and life expectancy as well as per capita income, Canada ranked number one. This gave many Canadians license to chide our American friends about their country's poor showing (merely number 12). UN officials repeatedly scolded Canadians because the index was never intended to be misused in this mannerit was designed to classify countries for development programs, not for propaganda purposes. (By the way, you don't hear a lot about the index around here anymore after Canada slipped to third place this year behind Norway and Australia.)
Just as countries can sometimes misuse classification models to lay claim to a superior quality of life, the software industry misuses classification models like the CMM to claim superior product quality.
The Software Engineering Institute's (SEI) Capability Maturity Model (CMM) is a five-step model for ascertaining the maturity of software development organizations. This score is determined with a survey of the company's compliance with the CMM's Key Practice Areas (KPAs). KPAs are what I would normally think of as industry best practices, including version control and defect tracking. The majority of software development organizations are level 1, which is considered random or undisciplined processes. Companies that follow minimal best practices such as defect tracking and version control move up a step to level 2. Institutionalize these practices, and you climb another step up the maturity ladder.
The CMM was originally developed to provide the government an objective scale to evaluate its contractors' software development maturity. It has become the means by which many in the software engineering field rate the maturity of their software development organization. It's a good model for evaluating your software engineering capabilities and identifying areas that you can improve on, but, just like the UN Human Development Index, it can be misused for political and commercial purposes.
Case in point: A while ago, I was talking to the development manager of a company that creates human resources management software, discussing the bids that I had received from different vendors for our e-commerce system, some of them in the million-plus region. He stopped short of telling me that I was insane to outsource to North American software houses, but enthused about his company's experience in outsourcing to India. For just 40 bucks an hour, he said, I'd have access to top-notch programmers.
I remembered that, in university, a lot of my friends were visa students from India. Many stayed in Canada, but quite a few returned home. They were good developers, and if I could hire high-quality talent for $40 an hour instead of the $175 an hour I was being quoted by most development houseswell, my friend was right: I'd be crazy to outsource to a North American software house.
My colleague introduced me to their outsource vendor's agent, a pleasant, enthusiastic chap who gushed about his company's technological prowess: leading-edge software technology, object oriented, RUP, Use Cases, UML, Rational Rose. And, about every fourth sentence, he was careful to mention that they were CMM level 5: To maintain their certification each year, they had to fill out the CMM survey and identify their implementation of the CMM KPAs to the auditor's satisfaction.
Though I view certification as a good sign, I'm always a little skeptical of a company that considers certification a guarantee of quality. A few years ago, I participated in a local corporate consortium about ISO 9000, another certification for process quality. Our favorite joke was "You could make lead life jackets and still be ISO 9000 certified." Also, I'm a little cautious about certification processes in generalfriends working in the defense industry have told me too many stories of being "coached" before filling out various quality audits.
My friend showed me a sample of the company's work products, starting with
use cases. They were all CRUD-type use cases, and I could classify them as either
Create X, Read X, Update X or Delete Xuse cases in name only. Though they
may have satisfied the CMM requirements, they certainly didn't delineate the
value of the system to its users. The class model was a poor combination of
massive "God" classes and data classes, with nothing more than
operations for every attribute. There was nothing object oriented about the
architecturea CASE tool like Rational Rose isn't a magic OO wand, and
automatic code generation from a bad model does not create great code. The use
of such tools may be a sign of software maturity, but it's no guarantee of product
Despite my concerns, my friend was happy with the vendor because he was receiving the benefits of a mature software development process with regular, predictable delivery of product at a defined level of qualityat a bargain cost. Unfortunately, sometimes you get what you pay for. With minimal software experience, my friend wasn't aware of the product's brittle architecture, nor did he anticipate the high cost of inevitable feature changes and enhancements in the future. As his product was designed to serve as the framework for a family of related products, this was a real disaster in the making.
With E-Co, I'd flirted with disaster long enough. When I told the vendor agent that we wouldn't pursue business with them, his entire counterargument was, "But we're CMM level 5!"
Well and goodbut I wanted more. Karl Wiegers wrote: "The CMM assumes that God gave you the requirements and you take them from there." My requirements specifications were anything but divinely inspired. I needed a vendor who would work with me to quickly resolve requirements issues, with the ability to recommend and implement improvements. The vendor's mantra "We're CMM level 5!" made me wonder if that certification would be used as a shield to deflect any complaints, framing product defects as E-Co's responsibility because of our imperfect requirements. I simply wasn't confident that this vendor had the abilityor desireto protect E-Co's interests.
While I strongly believe in a disciplined software development process, a high CMM level does not necessarily imply high-quality products. Great configuration management means that you'll always know what went into a build, but it won't improve the flexibility of the software in that build. Great defect tracking means that you probably won't ship software with the same defect twice, but it won't guarantee a great solution for that defect. To produce high-quality software, you still need a high-quality software development team.
Moral: There are lies, damn lies and maturity models. Beware of attempts to use the CMM beyond a statement of software development process maturity.
Next: First Flight