In This Issue
- System Integration on Large Agile Teams
- Book Review: Adrenaline Junkies and Template Zombies
- Hot Links
System Integration on Large Agile Teams
In my July 2008 Dr. Dobb's Journal column Agile and Large Teams I described how to scale agile teams to several hundred people in size. In that article I discussed the need for coordination by project leaders, product owners, and architecture owners to ensure that the large-scale project team works effectively. I also discussed the need for an independent test group, and how one of its responsibilities is system integration testing of the entire product being developed. What I didn't discuss is how system integration actually occurs on large agile teams, a topic that I now address in this newsletter.
On small agile teams integration is fairly straightforward. As an agile team gets larger, particularly when it is organized into a team of teams, integration becomes more complex in proportion to the complexity of the system being developed and the complexity of your organization structure. In very complex situations you may require one or more people to focus on the integration of the overall product and on supporting the integration efforts of the various subteams within your overall project team. This role is often called Build Master or Integration Manager.
To understand where integration fits in, let's work through the lifecycle of a large-scale agile project. At the beginning of the project, known as Iteration 0 or the Inception phase, part of your project initiation efforts includes architecture envisioning. With a bit of agile modeling you will identify the components/subsystems of the overall system that you're building, enabling a "system of systems" approach to your solution. This is critical because you want to organize your team around the architecture with each subteam being responsible for one or more of the subsystems. This in turn enables you to scale your project team to dozens if not hundreds of people.
Part of the initial architecture effort is to specify the initial interfaces to the individual subsystems. This enables you to decouple the individual subsystems from one another, and thereby reduce the need for coordination between the corresponding development teams, as much as possible. By definition you'll never do away with coupling between subsystems completely, and similarly nor will you completely remove the need for the teams to interact with one another, but you'll be able to simplify things dramatically. Once the interfaces are well defined it is possible for the individual subteams to focus on implementing the innards of those subsystems. The interfaces will evolve over time, hence the need for architectural coordination efforts throughout the project, as your requirements and understanding of the solution space evolve.
During the first development iteration, called the "Elaboration phase" in the Unified Process, an important goal is for the subteams to create mocks of their subsystems according to the interface specifications agreed to earlier. The goal is to do a complete, end-to-end build of the mocked out system to ensure that the subteams are working to the same technical vision. You'll undoubtedly discover that you need to evolve the interfaces a fair bit at this point as you run into technical issues that you hadn't thought through during Iteration 0. Evolving design is normal and should be welcomed.
Advanced teams will also implement technically risky requirements at this point in time, an effort that may last several iterations depending on the complexity of what you're building, to build an end-to-end working skeleton of your system. This is a critical risk-reduction technique because you effectively prove that your architecture works via running code. Some people think that this goes against the agile approach of having stakeholders prioritize requirements, but the reality is that there are both technical as well as business stakeholders involved with any project. My experience is that most technically complex requirements also provide high business value, so it really isn't much of an issue in practice. Proving the architecture early is one of the fundamental practices that RUP socialized within the development community in the mid-1990s, long before the agile buzzword was coined.
There are several advantages for making mockups of the subsystems available early in the project. First, the individual subteams can now move forward on their own work with few dependencies on the other subteams. Each team will evolve their subsystems throughout the project, replacing the mocked out portions of code with real working code. These new versions are made available to the other subteams, who in turn will choose when they want to integrate these new versions into their own environments. In my experience earlier is better than later but you want to wait until you know that new versions are stable, one of the advantages of having an independent test team, until you integrate it into your own work. Second, your independent testing team can now test against the entire build as they see fit. Granted, at first they only have mockups of the system but they can at least start organizing their test framework(s) for the system at this time. Third, you can similarly put together your integration framework to support your continuous integration efforts across the entire system as well as integration efforts on individual subteams.
Any reasonably complex integration effort will need tooling support. CruiseControl is great for simple-to-medium complexity but after than commercial tools such as BuildForge are often a better bet because of their greater robustness. In highly complex situations not only do you need to fully automate your system builds you should also automate the deployment of artifacts as much as possible. It should be easy for teams to share the working builds of their subsystems with each other, to deploy them into demo environments, into testing environments, and even into production environments.
Individual subteams will promote their code after they've tested it within their own environments. On large-scale agile teams these new subsystem builds are often vetted by your independent testing team before they're made available to the other subteams, a process that should be done quickly and often. The test team will often do a full system integration test, something that may be difficult for subteams to do due to timing considerations (integration tests often take a long time to run) or due to resource restrictions (the test team typically has a more sophisticated platform to test on). Subteams may choose to use pre-approved subsystems at their own risk, depending on your organization's culture.
You want to integrate, and better yet test, your overall system as often as possible. You also want to do so as early in the lifecycle as possible, once again underlying the importance of the initial architecture envisioning and initial subsystem mocks. This reduces the feedback cycle with respect to finding integration problems, reducing your overall costs associated with addressing those problems and very likely shortening your development lifecycle -- how many large systems have you seen look months at the end of the project once they run into integration problems?
In summary, system integration is an important aspect of any agile project, but it's absolutely crucial for large agile teams. To address the risks associated with such efforts you need to do some initial architecture envisioning at the beginning of the project to set your technical direction, mock out the identified subsystems right away, and set up your integration and testing frameworks based on these mockups. This will help to decouple your architecture as well as reduce the coordination needs between teams.
Book ReviewJunkies and Template Zombies:Understanding Patterns of Project Behavior
Tom Demarco, Peter Hruschka, Tim Lister, Steve McMenamin, James Robertson, and Suzanne Robertson
Dorset House Publishing, 2008
In January I attended a keynote speech given by Tim Lister and Peter Hruschka, two of the authors of Adrenaline Junkies and Template Zombies: Understanding Patterns of Project Behavior, where they overviewed a dozen of the 86 behavioral patterns which they'd written about in this book. About 10 minutes into their speech I decided that this was a "must read" book that I had to have. Sure enough, having now read the book cover-to-cover I can safely confirm that my initial reaction was correct. More importantly, my intuition now tells me that this is one of my "must keep" books which I'll refer to over and over again in the years to come.
This book describes both behavioral patterns, good things to do, and behavioral anti-patterns, bad things to do. Interestingly, both of the "patterns" in the book title are anti-patterns: Adrenaline junkies are "heroes" who take on too much work and who end up becoming bottle necks because only they can make critical decisions and template zombies are people focused on filling in the official standard templates regardless of the practical value of the content or of what is actually needed. Most of the patterns are in fact positive strategies, such as Eye Contact which stresses the important of distributed workers meeting together physically on occasion to build trust and communication channels. Young Pups and Old Dogs describes the differences, and the values of, staff members with varying levels of experience.
The authors have given names to these common behaviors that they have seen in IT organizations around the world. At times the behaviors described seem fairly common -- they are, which is why they're captured as (anti-)patterns. This is good, because as Mark Twain pointed out common sense isn't very common, so pointing common behaviors out to people can be quite valuable. Also, what you may think is common may be an eye-opening revelation to someone else, and vice versa.
The book is short at 237 pages. It is very well written and easy to read. The 86 behaviors are described in individual sections, making it easy to read a bit at a time or to use as a reference manual for years to come. For me it was a great airplane read. Anyone serious about their IT career will find this book to be a valuable investment.
My July 2008 Dr. Dobb's Journal column Agile and Large Teams describes how to scale agile teams to several hundred people in size.
Lean Development Governance, an IBM whitepaper that I co-wrote with Per Kroll, describes strategies for governing agile projects effectively. One of the practices is to organize your team around the architecture of your system.
My December 2007 Dr. Dobb's Journal column, Scaling Test Driven Development, describes how to enhance TDD with continuous independent testing.
My March 2008 Dr. Dobb's Journal column, Scaling Scrum, describes strategies for extending Scrum to meet the real-world needs most organizations face today.
The article Architecture Envisioning: An Agile Best Practice describes how to go about initial architecture modeling on an agile project.
See the Rational Unified Process (RUP) product page.
Stop by my Agile@Scale blog.