Dr. Dobb's | Scaling Scrum

Scaling Scrum

Scrum is an agile methodology that focuses on a subset of project management and requirements management. But some organizations are finding it can be a challenge to scale Scrum to meet the complexities of their real-world environments.

April 08, 2008
URL:http://www.drdobbs.com/architecture-and-design/scaling-scrum/207100381

Scott is a DDJ Senior Contributing Editor and author of numerous IT books. He can be contacted at www.ambysoft.com/scottAmbler.html.

Scrum is an agile methodology that focuses on a subset of project management and requirements management. Scrum has become popular in the past few years with the rise of agile software development, and is often used to round out Extreme Programming (XP); or perhaps XP is used to fill out Scrum, I guess it depends on your point of view. Scrum is particularly suited for "agile in the small" projects due to its simplicity, but as many organizations are finding, it can be a challenge to scale Scrum to meet the complexities of their real-world environments. The good news is that over the past few decades various people have identified a wealth of techniques which we can adopt and adapt to address these challenges.

Scrum was developed by Jeff Sutherland, John Scumniotales, and Jeff McKenna in 1993 during their work at Easel Corporation. It was then popularized through the efforts of Ken Schwaber and more recently by the Scrum Alliance (www.scrumalliance.org). Although there is a lot of material available online and in books about Scrum, my favorite resource is Mike Vizdos's www.implementingscrum.com site because it not only provides real-world advice for making Scrum work in practice, it does so in a light-hearted manner through the use of cartoons.

Figure 1 depicts the Scrum Construction lifecycle, modified from the lifecycle diagram presented by Jim Highsmith in Agile Software Development Ecosystems (Pearson Education, 2002) showing the fundamentals of the methodology. Scrum teams work from a product backlog, a prioritized stack of requirements maintained by the "product owner" who is responsible for working closely with the team to represent the overall stakeholder community. At the beginning of each "sprint", a timebox which is suggested to be 30 calendar days but can be any length in practice, the team identifies the requirements that they will implement during that sprint and plans out how they'll do it. At the beginning of each day throughout the sprint the team gets together for a 15-minute "scrum meeting" where each person indicates what they did yesterday, what they think that they'll do that day, and whether anything is blocking their progress. At the end of the sprint, the team should have produced a potentially deployable version of the system that they're developing which now implements the requirements that the team committed to at the beginning of the sprint. A demo is held with relevant stakeholders to obtain feedback and commitment to fund the next sprint. Many Scrum teams, as is the norm with agile teams in general, will hold a retrospective to identify potential improvements to the process that the team is following.

The lifecycle of Figure 1 is reasonable from a high-level point of view, but it leaves out a lot of detail. Scrum doesn't provide any advice for how to go about actual implementation; that's a detail that's left up to you. Nor does the lifecycle indicate how to initiate a project, or deploy your system into production at the end of the lifecycle. Recently Scrum has been touted as a process framework, but as you'll see a more accurate way to think about it is to view it as one part of the overall process picture. It is definitely a process framework from a consultantware point of view, and we've not only seen a consulting cottage industry grow up around Scrum but even an alliance of said consultants. I've found that to scale Scrum to meet the real-world needs of IT departments, you must consider the full development lifecycle and consider the full range of IT concerns.

[Click image to view at full size]

Figure 1: The Scrum Construction Lifecycle. Deliver a potentially shippable system every 30-day sprint.

The Full Development Picture

Construction is only one part of the overall software development lifecycle. Figure 2 presents an overview diagram of the Agile lifecycle which encompasses the ideas presented in Figure 1 and extends them to address the full range of software development issues. This lifecycle explicitly includes the concept of phases because the activities that you perform do in fact change throughout the span of a project. I've adopted the normalized terminology described in the sidebar plus other common agile phase terminology—many agile teams refer to project initiation effort as "iteration 0" and transition as the release phase. In the Eclipse Way, the Agile process followed by the Eclipse development team, they use the terms "warm up" and "end game" respectively. In Agile and Iterative Development (Addison Wesley, 2004), Craig Larman uses the terms "pre-game" and "release" for these phases. The Open Unified Process (OpenUP) uses the terms Inception and Transition respectively. Sadly, many people consider the concept of phases to be antithetical to agility, yet in practice they seem to be a fundamental lifecycle concept.

Figure 2 encompasses several critical activities to scale Scrum to meet your real-world development needs:

You need to do some initial requirements and architecture envisioning at the beginning of the project to address basic questions around scope, cost, schedule, and technical strategy. Few projects are funded without acceptable answers to these issues.
To ensure the quality of your work, you should have an independent test team that works in parallel to the development team. This team reports change stories back to the team who then treats them exactly like a requirement—they're estimated, prioritized, and put on the work item stack (see "Scaling Test-Driven Development", www.ddj.com/architect/205207998).
The "work item stack" contains all of the potential items of work that a team may need to address, not just stakeholder requirements. For example, you need to set up people's workstations, fix defects, people attend training, they take vacations, they review the work of other teams, and so on. These are important activities that must be planned for accordingly.
Iterations can be any length, not just 30 days, and an individual team needs to determine which is best for them. As Dr. Dobb's 2007 Agile Adoption Survey (www.ddj.com/architect/200001986) found, the most popular iteration length is two weeks.
Effective agile teams prove their architecture with code early in the project (this is the Elaboration part of Elaboration and Construction). Although purists claim that you should prioritize based solely on business concerns, the reality is that it behooves you to address technically risky requirements early in the lifecycle. Luckily, business value and technical risk often go hand-in-hand, so you rarely need to "game" your work item stack much to accommodate this strategy.
The Transition phase provides time to successfully release software into production. This effort includes training people, finalizing testing, finalizing documentation, pilot/beta testing, and deploying the system; see "The Agile End Game" (www.ddj.com/architect/198800565) for more detail.
Enhancement requests and defect reports are identified once the system is in production and treated as new work items that must be prioritized.

[Click image to view at full size]

Figure 2: The Agile Software Development Lifecycle. The full development lifecycle must address project initiation, construction, and deployment.

Since the late 1990s, I've written that effective software development is serial in the large, iterative in the small, delivering incremental releases over time. Over the past few years the rhetoric around agile software development has unfortunately drowned out the "serial in the large" part of the message, forcing many people to rediscover proven techniques and strategies. If the lifecycle of Figure 2 makes sense to you, and you'd like to extend Scrum to address these issues, I suspect that you'll find that the OpenUP, downloadable free of charge at www.eclipse.org/epf, will be of interest to you.

Enterprise Concerns

Most IT departments have numerous systems in production, many currently in development, and many more under consideration. The implication is that to be truly effective you need to take the cross-system IT lifecycle into account, not just the system lifecycle. Yes, letting your architecture evolve over time is a great concept, but wouldn't it be great to take advantage of a common architecture and reusable assets? Yes, we want to implement requirements in priority order to achieve the highest return on investment (ROI) possible for that system, but wouldn't it be nice to work on the most important projects to begin with? Yes, it's nice that we can mock out access to data sources, but wouldn't it be great to easily work with existing data sources so that we don't have to create yet another silo database? Yes, retrospectives are great ways to identify potential improvements for a team to adopt, but shouldn't we have a mechanism to share those findings with other teams? We need to look beyond the needs of single project teams; otherwise, we've merely found ways to more efficiently create silo application, contributing to the overall maintenance burden within your organization.

You can, and should, be as agile as possible at these activities, and yes, you can choose to do so. A few years ago, I wrote The Enterprise Unified Process (Pearson Education, 2005) with Mike Vizdos, which described how to address the range of enterprise-level issues in as agile a manner as possible. The goal of these enterprise activities should be to support the overall organization as well as to enhance the efforts of project teams. The good news is that we're seeing greater focus on enterprise-level issues within the agile community. The bad news is that we seem to be reinventing the wheel. We don't need to because there's a lot of great material out there if we only choose to leverage it. The EUP clearly takes a lightweight, collaborative approach to enterprise activities, and the Information Technology Information Library (ITIL) v3 provides even greater detail if you're interested. You can find out more about the EUP at www.enterpriseunifiedprocess.com and about ITIL at www.itil-officialsite.com.

Agile@Scale

There's a lot more to software development than the construction lifecycle of Figure 1. Let's not get fooled by marketing rhetoric and let's not think that we need to reinvent the wheel just because we have a new generation of methodologists who are in the process of figuring things out yet again. There's a lot of value in Scrum, but we need to approach it with open eyes and recognize that it's only one part of the overall IT process solution.

Normalizing the Terminology

Scrum uses some strange terminology, a strategy that has its advantages and disadvantages. The primary advantage is that it helps people new to agile development to break out of their preconceptions as to how things should work. Yet this comes at a price. People initially find the terminology to be confusing because it really doesn't make much sense. For example, nobody "sprints" through a long-distance race. The term "Scrum Master" instantly evokes thoughts about someone wearing leather who whips people if they don't do what they're commanded, the exact opposite of what the role is really about. The term "Scrum meeting" makes no sense whatsoever to people who aren't familiar with rugby, and arguably "team huddle" or simply "daily stand-up" meetings are better terms. Worse yet, anyone who has ever played rugby will tell you that scrums are only fun up until the point that you get elbowed in the face. In short, more accurate terminology would be:

• Iteration or Time Box instead of Sprint.

• Coach or Team Leader instead of Scrum Master.

• Daily Stand-Up Meeting instead of Scrum Meeting.

—SWA