Is software engineering still relevant?
For the July/August issue of IEEE Software, Tom DeMarco, a principal of the Atlantic Systems Guild, wrote a “Viewpoints” essay entitled “Software Engineering: An Idea Whose Time Has Come and Gone?” It’s a well-written piece which essentially repudiates all his earlier work supporting software engineering. And why? The short answer is economics and cost/benefit analysis. His earlier work promoted the idea that you can’t manage what you can’t measure so software metrics are good. He now says that software metrics cost too much time and money. Ultimately, his essay plays to two popular programming and management fads:
Team programming. which eliminates entire layers of mid-level management and their associated costs, is good.
Incremental programming practices (e.g. agile development, extreme programming, etc.) are good.
He presents his arguments, not as recommendations for software development, but as a way to manage software development. He makes the following statements:
“Can I really be saying that it’s OK to run projects without control or with relatively little control? Almost. I’m suggesting that first we need to select projects where precise control won’t matter so much. Then we need to reduce our expectations for exactly how much we’re going to be able to control them, no matter how assiduously we apply ourselves to control.”
“So, how do you manage a project without controlling it? Well, you manage the people and control the time and money. You say to your team leads, for example, ‘I have a finish date in mind, and I’m not even going to share it with you. When I come in one day and tell you the project will end in one week, you have to be ready to package up and deliver what you’ve got as the final product.’”
“Consistency and predictability are still desirable, but they haven’t ever been the most important things.”
Wow – I wish I had the luxury of working in his world! I spent most of my career working in industrial embedded systems. In my world, software/firmware’s functionality was often preordained by industry or international standards or mandated by law. I rarely had the luxury of incremental programming – it all had to work as specified, otherwise there would be no product. And if the code I produced wasn’t “consistent and predictable”, bad things happened: people could die, vast millions of hard dollars could be lost, irreparable environmental disasters could result. Granted that these fads are primarily artifacts of desktop business application programming where failures may, at most, cost some soft dollars and some embarrassment. Still, that doesn’t mean they’re not worthwhile goals even there.
There was an excellent book which went largely unnoticed a wile back, Scott Adams’ The Dilbert Principle. While most marginalized it as a humor book, it had a serious message. A watershed moment in management history was the publication of The Peter Principle which stated the now well-known idea that in a hierarchy, individuals will be promoted to their level of incompetence, where they will remain. Ultimately the organization will be filled with people who are incompetent to do their jobs. The immediate aftermath of The Peter Principle was the rush to acquire MBA degrees and replace these “incompetent” managers with trained professionals. The Dilbert Principle pointed out the fallacy – in the old order, everyone in the organization was at some point in their career competent to do something. In the new order, the only proven management competency was the ability to earn an MBA.
Nowhere is this truer than in engineering. When I first entered the field (before the publication of The Peter Principle), most engineering departments had mid-level technical managers. In addition to managing, they were the technical gurus who assigned projects, built teams, and were responsible for their group’s budget and schedule performance. Then, almost overnight, they were gone as a budget-cutting move. Now, teams were constituted on a largely ad hoc basis by managers with far less technical insight, senior engioneers with minimal management experience, and/or even HR personnel. But when failures began to mount, no one ever thought to question the structural changes that preceded them. After all, these were all changes instigated by trained MBA professionals.
OK, enough ranting… What’s my take on the subject of software engineering?
Software is unique. It’s the only engineering discipline where the thought process becomes the product with little intermediate translation. Still, it is engineering and you ignore that at your peril! As with any other engineering field, poorly practiced, it can lead to disaster – even if only financial disaster.
So what differentiates software engineering from simple programming? The inherent answer is discipline – the discipline that is required of all engineering fields. In practice, this translates to rigorous design and testing. As DeMarco noted, metrics can be a source of contention, but if you only use one metric, then use modified complexity (like cyclomatic complexity, but with each switch statement counted as only a single path) as a predictor of maintainability. Most importantly, take the time to actually design your project before you actually start coding it. When insufficiencies are detected, don’t hack code, but go back to the design. Change the design and the code will follow. Any time I see a project where the design time is significantly less than 40-50% of the total development time, I smell a potential disaster. Likewise, when I see software staff cave into clueless management demands to look busy and start coding without a design in place, disaster is usually sure to follow. It’s just that simple – and just that difficult.
Software engineering isn’t an idea whose time has passed, but one which is even more essential as we forget the lessons that brought it about in the first place.