Software development is very hard for a host of reasons. Sadly, many writings about software development provide advice for simple situations, glossing over the real challenges that we face in practice. This is akin to sticking our heads in the sand It is time to raise our heads up and recognize the complexity of the situation that we face.
In early 2014, we ran a State of the IT Union survey to help identify the range of situations currently faced by software development teams. The survey had 231 respondents: 70% of respondents had 10 or more years of experience in the software development field, 42% worked in North America, and 47% in Europe. A bit more than half (52%) work on teams following a primarily Agile approach, 19% on iterative teams, 13% on ad-hoc teams, 8% on traditional teams, and 7% on lean teams.
Not surprisingly, the survey found that software development teams are working in a very wide range of situations. There are small teams, medium-sized teams, and very large teams. Some teams are co-located, but even more teams are geographically distributed in some manner. Some teams face relatively straightforward development challenges, but in most cases, teams face significant technical and domain complexities. Some teams face regulatory situations. All these factors team size, geographic distribution, organizational distribution, domain complexity, technical complexity, and regulatory concerns have a great impact on how teams work and how they organize themselves.
Some Software Development Teams Are Large
The survey found that the majority of teams, 58%, are small (10 people or less); 36% of software development teams have between 11 and 50 IT people in them; and the remaining 6% of teams have more than 50 members. When it comes to Agile teams, only 52% comprise 10 people or less, which is not what I expected given the general Agile advice on keeping teams small. Agile teams may be a tad bigger than non-Agile teams on average because Agile teams are more likely to be working on harder problems (more on this later).
Larger teams require greater communication and organizational overhead and thus have greater risk than smaller teams. For example, coordination within a small Agile team may be addressed by a daily stand-up meeting and a biweekly detailed planning session. A medium-sized team may require two levels of stand-up meeting, a "scrum" and a "scrum of scrums," as well as the planning session. A large program, organized in five or more subteams, requires far more sophisticated organizational structures (see Agile and Large Teams) to address architectural evolution, requirements management, and (egads!) project-management issues. In short, the complexity of your coordination strategy increases as your team size increases, thus increasing the overall difficulty within the team to work together effectively.
Most Teams Are Distributed
The survey asked about two forms of distribution: geographic distribution and organizational distribution. Survey respondents reported that 70% of their teams are geographically distributed in some way: A team that has people working in cubes or offices, even on the same floor of the same building, suffers from the effects of geographic distribution. Worse yet are teams that have people are working from home, different cities, or even different continents. Of the teams that are geographically distributed, over half included people working from very distant locations. The good news is that only 61% of Agile teams are geographically distributed, an indication that the Agile advice regarding the importance of co-location is being heeded to some extent.
A few years ago in The Distributed Agile Team, I explored the difficulties faced by distributed teams. Teams that are geographically distributed suffer from a range of communication and organizational risks that must be mitigated by a combination of improved tooling, coordination activities, and travel. When someone is sitting right beside you, it is easy to have a quick conversation with them to get some help or to discuss an idea. The farther away someone is physically, even when they are just down the hall from you, the harder it is to have such conversations. More difficult yet is when the colleague is in another time zone (because the greater the time zone difference, the greater the difficulty in scheduling such conversations). Geographic distribution motivates you to use less effective modes of communication (such as email, online chat, or video conferencing instead of face-to-face communication) increasing communication risk within your team (see Communication on Agile Teams for some insights). When you're sitting in the same room as your teammates, you will know what each other is doing, and whether it's going well, through osmosis. When you're separated by cubicle walls, by real walls, or worse yet, by thousands of kilometers, the "simple" task of keeping up with everyone else becomes very challenging. Team dashboards, wikis, shared document repositories, and status reports become important lifelines in such situations.
Two thirds of software development teams, regardless of paradigm, are organizationally distributed in some manner. For example, 27% include people from different divisions within the company, 20% include employees from other countries, 39% include contractors, 13% include people from other partner organizations, and 21% of teams are working with outsourcers. Organizational distribution increases communication, political, and contractual related risk to the team.
When it comes to outsourcing, it is my experience that problems stemming from your procurement strategies are quite vexing. This is particularly acute for Agile teams because procurement processes often reflect traditional thinking. An Agile procurement process sets up a collaborative, not confrontational, relationship. When outsourcers are involved, they should follow an incremental delivery stream where every week or two they demo the working solution that they've built to date, and their progress should be measured by actual delivery of working code. This is very different from the traditional strategy of producing a (often fictional) status report each week leading up to an (often late and buggy) deliverable toward the end of the lifecycle. Agile practices such as providing continuous visibility into everyone's work, incremental delivery, and collaborative work become very difficult when the contract doesn't specifically spell this out from the beginning. Even when you have an Agile contract in place, it can still be difficult: Team members might not know what information they should share with non-employees, so you have to put mechanisms in place to maintain rights to your intellectual property (IP), and communications between the organizations involved are rarely as streamlined as they need to be.
Most Teams Face Great Complexity
The survey explored the level of technical and domain complexity faced by software development teams. Regardless of paradigm, 90% of teams face some form of technical complexity, such as working with legacy assets, developing for multiple platforms, or working with packages. These technical complexities are typically driven by the organization's existing infrastructure and are often not within the scope of control of the team. Interestingly, 83% of teams faced technical debt issues such as working with or, better yet, fixing legacy software or data sources. Technical debt increases the technical risk on software development teams, reducing their ability to deliver new functionality in a timely manner. One in ten software development teams were performing a package implementation, also known as a commercial-off-the-shelf (COTS) implementation.
Teams facing technical complexity find that they need to make a greater investment in architecture and design strategies. This doesn't necessarily imply that they need to create detailed documents, although in a few cases it might, but it does imply that they need to think through the technical aspects of the solution. For example, technical unknowns such as how to work with a new technology or how a combination of technologies work together may need to be address via architecture spikes (called architecture prototypes outside of the Agile community). Somewhat more effort may be spent in up-front architecture modeling, spending a few hours in simple situations compared with a few days in more complex situations (note that this is work time and not calendar time that I'm quoting). Detailed design issues will typically be addressed via a combination of model storming and, at least on disciplined teams, test-driven development (TDD). The team requires a more sophisticated approach to testing to deal with the greater risk of defects posed by technical complexity, perhaps requiring a parallel independent test team in addition to their own whole team testing efforts. All these practices are important because they enable teams to reduce technical risk, but they require a greater skillset from team members and more sophisticated ways of working.
When it comes to domain complexity there was a difference by paradigm. In all, 50% of respondents indicated that their teams were addressing complex or very complex domain problems. For Agile teams, 61% reported facing such domain complexity levels, compared with 39% for non-Agile teams. This shouldn't be a surprise, as the highly iterative and collaborative strategies of Agile are an effective way to address the risks of greater domain complexity.
Teams facing domain complexity will adopt more sophisticated ways to elicit requirements from their stakeholders than teams with simpler challenges. Their initial scoping efforts will likely be more involved, it will likely require a wider range of artifacts and may even result in more detailed specifications. Greater collaboration with stakeholders throughout construction will also likely be required, perhaps moving away from Scrum's Product Owner strategy to the more refined approach of active stakeholder participation from Agile Modeling. Greater domain complexity requires more complex approaches to testing, both on the process side and the tooling side, similar to dealing with technical complexity.