In This Issue:
- The Astronomy Analogy for Software Development
- Book Review: How to Measure Anything
- Hot Links
The Astronomy Analogy for Software Development
One of the difficulties of helping organizations improve the way that they work is that the people who are asked to change are often unwilling to do so. Sure, they recognize that the existing process isn't perfect and they have some suggestions for ways to improve things, but many people will hold to the mistaken belief that the underlying principles of their approach are solid. When I explain the basics of agile software development they're often surprised that the agile community has abandoned many strategies that traditionalists believe to be "best practices". In their shock and dismay they'll exclaim that surely the laws of physics must still apply as they retreat back into the comfort of their traditional shells. Yes, the IT laws of physics apply regardless of the development paradigm, but the point that I want to make in this update is that it's doubtful that we know what those laws actually are. So I'd like to share what I call the astronomy analogy for software development to explain how our understanding of the laws of IT physics is clearly still evolving.
Consider how our understanding of astronomy has changed over time. Around 500 BC Pythogoras posited that the sun, moon, other planets, and stars revolved around the Earth on crystalline spheres. This theory was comforting to people, it made sense because it was easily observable because we could see how things moved through the sky, and surely no one could question that the Earth was the center of the universe.
Two millennia later, around 1540, Copernicus published his theory that the planets revolved around the sun. The man was clearly a heretic and the traditional astronomical community scoffed at his ideas. Then in the 1600s another heretic by the name of Galileo came along and extended Copernicus' new paradigm. To address this challenge the Catholic church subjected Galileo to the Inquisition, it was far easier to deny the evidence before them than it was to change their way of thinking, and Galileo was locked up for spreading his ideas. Astronomy had a new paradigm, albeit one not adopted by everyone, that showed that with the exception of the moon everything that pretty much mattered revolved around the sun and not the Earth. Our understanding of the astronomical laws of physics had undergone a major paradigm shift.
Then a few decades later yet another heretic came along, Isaac Newton, who in 1687 published his laws of motion in Principia Mathematica which explained, amongst other things, the fundamentals of how moving bodies interact with one another. At that point is was very clear that the universe didn't revolve around the sun but instead that all bodies affected one another through gravitational forces. Our paradigm had shifted yet again. Then in the early 1900s Albert Einstein introduced us to the laws of relativity, tweaking our understanding even more, and even today astronomers are still exploring the fundamental laws of the universe.
Now consider the history of IT. In the 1960s, at the prompting of the U.S, military, we decided that software development needed to be an engineering discipline. The theory was that building a software-based system was just like building a bridge, or perhaps a sky-scraper or some other impressive building, and therefore we needed to adopt processes which were similar to those in other engineering disciplines. This theory was comfortable to people, it made sense because it was easily observable that software development was as hard if not harder than building bridges, and no one could question that we couldn't benefit from greater engineering rigor. Just as the ancient Greeks had decided that everything revolved around the Earth we had decided that everything would revolve around engineering practices.
Then in the 1980s the heretic Barry Boehm introduced us to the Spiral model of software development, an iterative and incremental paradigm for development which was clearly different than the serial engineering model. The 1990s saw the introduction of other iterative methods, including Rational Unified Process (RUP), Scrum, Dynamic System Development Method (DSDM), the Object-Oriented Software Process (OOSP), and Extreme Programming (XP). 2001 saw the publication of the Agile Manifesto which marked the official beginning of the Agile movement, although unofficially it had started years earlier. The second major software development paradigm, iterative/agile, was upon us. Just like Renaissance Europeans had decided that most things revolved around the sun, modern developers have decided that software development projects should revolve around them.
The point is that even with the agile software development paradigm we still haven't gotten it right. The first paradigm, the traditional serial software engineering strategy, clearly didn't fulfilled its promise. Traditional strategies have resulted in low quality silo applications, one merely has to look at legacy code and legacy data sources to see this. The iterative/agile paradigm has fared a bit better, the greater level of discipline and focus on quality techniques such as regression testing and refactoring have produced higher quality applications, albeit ones that are more often than not silo applications. Just like our understanding of astronomy moved beyond the belief that everything must revolve around the sun, our understanding of effective IT must move beyond the belief that everything revolves around developers.
The third IT paradigm, which I suspect we may not see in our lifetimes, should be based on taking a disciplined enterprise view which takes the full range of issues into account. Although the traditional community arguably has a better understanding of issues surrounding enterprise architecture and portfolio management, they don't seem to have a viable strategy for acting on this understanding. The command-and-control, bureaucracy-heavy approaches of traditional enterprise methods simply don't reflect the complexities of the intellectual work inherent within IT. The agile community certainly doesn't understand these issues, being too wrapped up in low-level programming techniques, XP machoism, and ethically questionable although very lucrative certification efforts. Agilists have become better at delivering systems, even ones which extend existing legacy assets, but their general disdain for architecture and the required coordination to address enterprise-level issues limits their ability to deliver high-quality, enterprise-class systems which fit well into the overall organizational environment.
It's great that we can deliver high-quality applications which meet the needs of stakeholders in a cost-effective and efficient manner. It's not so great that we're putting yet another stand-alone database into production containing data which already exists in numerous other production databases. It's not so great that the system uses yet another new technology which the operations department now needs to support. It's not so great that the application implements business rules which have already been implemented by other existing applications, and worse yet that the implementation of some of those rules is inconsistent with previous implementations. As IT professionals we need to do better and scale agile approaches to the enterprise level.
So, what should this new paradigm shift look like? We should clearly keep the discipline and flexibility inherent in agile approaches, but need to extend them to address the full system lifecycle from cradle-to-grave, including initial "iteration -1" planning as well as operations and support, instead of just the "sexy parts" of the development lifecycle. Furthermore, we need a multi-system IT lifecycle, not just a single system lifecycle, because we work in organizations with a multitude of systems which affect one another. To address cross-system technical issues we need effective approaches to enterprise architecture and reuse which enables teams to easily leverage existing assets and to integrate easily into the overall infrastructure. To address cross-system business issues we need effective approached to portfolio management and to business modeling so that we can invest in the right projects and then govern those projects effectively so that we continue to invest in the right projects and abandon the projects which prove otherwise. To run our systems effectively we need effective approaches to operations and enterprise administration which enhance development efforts instead of hinder them, and vice versa. In the real world, there's a bit more to being successful at IT than just banging out working software on a regular basis.
The good news is that a few years ago Mike Vizdos, of Implementing Scrum fame, fleshed out my earlier work on the Enterprise Unified Process (EUP) to address exactly these sorts of issues. It extends iterative/agile processes such as RUP, Scrum, and XP to address the full lifecycle of systems as well as enterprise issues in a disciplined, collaborative manner. EUP takes the best of traditional enterprise thinking, iterative/agile thinking, but abandons the command-and-control bureaucratic strategies of the traditional world. My points are that it is possible to address these issues effectively if you choose to do so, and that we have an existing "reference process" to leverage if so.
The bad news is that the disciplined enterprise strategy of the EUP requires greater process maturity than most organizations are currently capable of. This need for greater maturity, an issue for any enterprise-level strategy and not just the EUP, is why I fear we're not going to see this next paradigm shift within our lifetimes. Just like it took astronomers decades to understand adopt the Newtonian paradigm, it will take the IT community decades to recognize the need for, and the fundamentals of, a disciplined enterprise IT paradigm. I hope that I'm right that we'll eventually make this paradigm shift, but I hope that I'm wrong about how long it will take us to do so. Time will tell.
Book Review: How to Measure Anything
How to Measure Anything: Finding the Value of Intangibles in Business
Douglas W. Hubbard
John Wiley & Sons, 2007
One of the primary challenges with managing and governing IT effectively is that many of the questions that we need to answer are difficult to measure. What is the expected value of a new software project? What is the chance of success? How long will the project take? What architectural strategy is best? How effective is a development technique? What is our level of quality? How good is our production data? And so on.
Although these questions are hard to answer, luckily Doug Hubbard provides in How to Measure Anything some proven advice for easily taking measures that enable us to improve our decision making.
To understand the value, and ease of, taking presumably difficult measures, in Chapter 2 Hubbard works through examples from past of great thinkers who didn't give up in the face of the "impossible". For example, around 200 BC Eratothenes estimated the circumference of the Earth by observing the lengths of shadows, Enrico Fermi estimated the power of the first atomic bomb by observing the distance that it blew confetti, and at the age of nine Emily Rosa (who became the youngest person to publish in a scientific journal at the age of 11) measured the ability (or more accurately lack there of) of people claiming to have the ability of therapeutic touch. Chapter 3 goes on to discuss the illusion of intangibles, motivating you to abandon the self-defeating belief that some things are just too hard to measure. Chapter 4 clarifies the measurement problem, focusing on uncertainty and risk, putting you in a better position to effectively reduce business risk through relatively simple measurement.
Chapters 5 through 7 describe more of the fundamentals behind measurements and the value of improved information, and chapters 8 through 10 describe strategies for doing measurements. Being a firm believer in strategies which reflect human behavior, I was particularly interested in chapters 11 through 14 which cover the human issues around measurement, making a hard science soft again.
If you're tasked with improving your internal metrics program, improving your governance strategy, or simply want to learn about strategies to find out what the heck is actually going on within your organization or industry then this book will prove to be a good idea. Hubbard uses straightforward, easy to understand examples throughout the book, thereby simplifying many complex ideas for the reader.
Gene Smith's A Brief History of Astronomy is a wonderful summary of the history of astronomy.
Agile Model Driven Development (AMDD) best practices describe strategies for effective modeling and documentation on agile projects, key aspects for scaling agile development.
The Enterprise Unified Process (EUP) describes how to extend the software development lifecycle to address the full lifecycle of a system and how to apply agile strategies to enterprise-level issues.
The Implementing Scrum site includes great articles and an online forum for discussing Scrum in an open manner. This is a great resource for anyone interesting in applying Scrum in a realistic manner.
My September 2008 column, Iteration Negative One, describes the portfolio management efforts which occur before the "start" of a software development project to select the right projects and get initial funding for them.
Agile Enterprise Architecture describes strategies for how enterprise architects can interact with, and effectively support, agile software development teams.
My Agility@Scale blog is here.
Please note that with this installment, the title of Scott Ambler's monthly newsletter is changing to "Dr. Dobb's Agile Update." Only the name is changing. The content will continue to be the great insight that Scott brings to the software development community. We apologize in advance for any confusion that might result. If you have any questions, please contact us at firstname.lastname@example.org.