Complex Requirements On an Agile Project

There's nothing simple about complex requirements and agile projects. Still with a discipline and a little luck...


October 31, 2008
URL:http://www.drdobbs.com/architecture-and-design/complex-requirements-on-an-agile-project/211800534

Scott is Practice Leader Agile Development for IBM Rational.


The real world is a complex place, resulting in complex requirements for any system that has to work there. This is true regardless of development paradigm. Although "agile in the small" methodologies such as Scrum and Extreme Programming (XP) have done much to show us how to improve our approach, too many people have thrown out the requirements management baby with the bureaucracy bathwater after putting too much faith in the overly simplistic strategies of those processes. Luckily, with a bit of discipline, it is straightforward to address the inherent challenges of complex requirements in an agile manner without resorting to the documentation-heavy practices favored by the traditional community.

Beyond Scrum

The Scrum method has popularized the idea of managing requirements as a stack of small, functional chunks, captured in a prioritized stack called a "product backlog". The idea is that at the beginning of each iteration/sprint, you pull an iteration's worth of work off the top of the stack. If only it were that easy. Although Scrum has helped us to get away from the onerous change prevention strategies (oops, I mean change management strategies) of traditional methods, it has blinded a generation of developers to the inherent complexities and nuances of understanding and implementing requirements. Scrum's product backlog concept works well for simple functional requirements, but as I described in "Beyond Functional Requirements on Agile Projects" (www.ddj.com/architect/210601918), it comes up short for nonfunctional requirements and architectural constraints. These types of requirements are addressed through three strategies: initial architectural/requirements envisioning, continuous independent testing, and education.

Not only do agile development teams implement new requirements during an iteration, they also fix defects and perform other nonrequirements work. Figure 1 depicts the OpenUP's (www.eclipse.org/epf) work management strategy, an extension to Scrum's product backlog strategy. The first thing that you should notice is that it's not just a product backlog but also a stack of work items. A work item is often a small functional requirement, such as a user story or feature, but can also be a defect or a nonrequirement item such as reviewing the work of another team, attending a training session, or setting up equipment. In "Scaling Test-Driven Development" (www.ddj.com/architect/205207998) I showed how disciplined agile teams will have an independent test team working in parallel to the team, validating new builds as they're made available, and reporting defects back to the development team. By treating defects as a type of requirement that gets prioritized, estimated, and put on the work item stack, you can simplify your change management process by in effect combining defect and requirements management into one. Furthermore, although you can try to plan for nonrequirement work items by fudge factors in your planning game efforts, the reality is that these fudge factors will vary between iterations because these sorts of work items don't occur as a steady stream throughout a project. To see this work item backlog strategy in action, you can download a trail version of Rational Team Concert (RTC) from www.jazz.net, which integrates this functionality into a Eclipse-based development environment for distributed agile teams.

[Click image to view at full size]

Figure 1: Work items are addressed in priority order on disciplined agile projects.

Another extension captured by Figure 1 is the explicit recognition that you should look down the work item stack an iteration or two for potentially complicated work items. When you find one, and they do occur, if it is close to being popped from the top of the stack, it behooves you to follow Agile Modeling's Model A Bit Ahead best practice (www.agilemodeling.com/essays/bestPractices.htm) and explore that work item and thereby avoid being blindsided by it. Although this can sound like heresy to agile purists, as we saw in DDJ's 2008 Modeling and Documentation survey (November 2008), it is quite common for Agilists to model. On the surface this may appear to go against Extreme Programming (XP)'s philosophy of having the courage to worry about tomorrow's problem tomorrow. Although this type of machoism might make developers feel good about what they're doing, and it is certainly an excellent juxtaposition to the risks inherent in detailed up front modeling, when taken to the extreme this philosophy proves to be an incredibly poor strategy in practice because sometimes complex work items can put your project at risk if you're not expecting them. Remember that nowhere in XP or Scrum does it say to be stupid, so don't be afraid to extend those methods with proven practices from other sources.

Actual Usage Is Complex

Complex functional requirements can clearly be organized into collections of smaller functional requirements, a process called requirements disaggregation, but that doesn't imply that those smaller requirements still have value outside the context of the larger collection. For example, you could write user stories indicating that your system should capture the name and address of a customer, that it should produce a mailing list of customers, that it should enable you to narrow down that list given certain criteria, and that it should output that mailing list in a standard XML format. All interesting requirements, but each one on their own provides very little value until you put them in the greater context of sending out marketing literature to a targeted group of customers. Until all of those user stories, and more, are implemented your system really doesn't provide true business value to your stakeholders, regardless of the fact that you have been producing potentially shippable software every iteration up to that point.

Figure 2 shows how requirements on a stack can be related to one another. For example, the red requirements need to be shipped together even though one of them is further down the stack than others. The green requirements are also related to one another, although they are clustered closer to each other than the red ones. Once the product owner recognizes that there are relationships between items on the work item stack, and that a lower priority item may be preventing you from releasing higher priority items and thereby lengthening the time to value for your project, they may decide to rethink their strategy. This rethinking might mean that they reprioritize the work items or that they renegotiate with the larger group of stakeholders that they represent and drop the dependency on the lower priority work item.

Figure 2: Small functional requirements only support part of the usage workflow that provides actual value.

Understanding these more complex requirements requires a more sophisticated approach than user stories or features by themselves. A simple strategy is to organize them into "epics" or feature sets respectively, collections of small requirements that are somehow related to one another. This is similar to throwing the requirements into separate buckets—it organizes them but doesn't provide context. Context is better addressed through usage scenarios or use cases. Many Agilists are leery of use cases because they've only seen heavy use cases, often because the business analysts got out of control and wanted to capture every possible detail in the use cases. But as I show at www.agilemodeling.com/artifacts/systemUseCase.htm, you can write very simple use cases in point form so as to think through the usage of your system without taking on the risk of needless documentation. Each of these bullet points could be addressed by zero or one user stories—the order of the use case steps and the "filler steps" that are addressed by manual activities or functionality implemented elsewhere provide the needed context that isn't captured by the individual user stories.

In "Scaling On-Site Customer" (www.ddj.com/architect/204801134), I argued that the role of XP's on-site customer—Scrum's product owner—is actually quite difficult because they need to represent a wide range of stakeholders, not just end users. This is why context is so important—different types of stakeholders will interact with the system in different ways, and you need to understand their various and often contradictory usage patterns and act accordingly. Sophisticated requirements artifacts such as use cases provide this context, the challenge then becomes writing use cases in an agile manner that avoids the needless bureaucracy often exhibited by traditional teams.

Requirements Cross Components

In "Agile and Large Teams" (www.ddj.com/architect/208700162), I described how large agile teams are organized into small agile subteams, often around the architecture of the system being built, and how the product owners of each subteam need to coordinate their efforts with the other product owners. The reason for this is depicted in Figure 3, which shows how a given usage requirement is implemented as a collection of smaller requirements implemented by different subteams. In the example, the yellow work items form a single usage requirement, with the lines between the work items depicting the logic flow of the scenario. The product owners on each subteam need to understand these dependencies between requirements and act accordingly. This coordination will occur via regular working meetings of all product owners, a "scrum of scrums" focused on requirements, as well as through impromptu sessions between individual product owners to iron out specific problems.

This is not only an issue within a single large system, it is also an issue between systems. Very few people sit down at their computer and use a single application all day long; instead, they use several in different ways throughout the day. The implication is that for the individual systems to add true value to the stakeholder, the required functionality of each system must be implemented. For example, assume that a banker uses a customer information system, an account information system, and a printing system to open a bank account for a person. Even though the new versions of the customer and account systems have been deployed into production and have the requisite functionality, the printing system has run into problems and won't be deployed for another three months. As a result, the new functionality in the other two systems will go unused, through no fault of their own, potentially making them appear to be failures. For several years now, IBM has acted on this realization and actively identifies green threads (www.ddj.com/development-tools/196603524), which are common usage scenarios that cross several systems, enabling us to better understand the needs of our customers and thereby deliver better product to them.

Figure 3: On large-scale agile projects usage scenarios cross several components.

Keep It Simple

Requirements often prove to be very complex in practice. Although agile methods, in particular Scrum, have offered good advice for simplifying requirements management, there is a distinct risk that their advice is overly simplistic and will put your project teams at risk. The advice of the traditional community, however, is overly complicated and bureaucratic, also putting your teams at risk. A middle road does however exist, as I've shown in this column, and you can address the complexities of complex requirements management without taking on the risks inherent in traditional change prevention strategies.

Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.