The main goal of software is to automate and simplify what would otherwise be accomplished using a manual process. This gives users of the software leverage to do more with less. Instead of balancing our checkbook by hand, we can use Quicken to do it faster and more accurately. Instead of maintaining the records of millions of people's financial transactions with paper and pencil, banks use mainframes.
Whatever industry you work in, what do you think about when you are creating new software? Don't you think about how it scales to meet your customers' needs as they grow with high availability? Even if you don't achieve that on the first try, aren't you still thinking about it and striving to achieve it? You know that to do it, you need to make the right technology and architectural choices. Over time, you may need to change some of those choices to keep pace with competitors. Even if your current needs are modest, software development architecture has evolved principles, patterns, and technologies to allow software to scale from a single user to thousands or even millions of users on multiple platforms at multiple locations with 99.999 percent uptime.
On the other hand, as an industry, we don't seem to be very good at scaling up the process of software development itself. Even when what needs to be built is primarily made up of well-understood tasks and requires little discovery, it seems that when the size of the effort goes beyond 5-10 people, things start to break down pretty rapidly. Adding more people usually doesn't scale in the same way that adding more hardware does.
This stems from the belief that software automates a well-defined problem but that the process of software development is a creative endeavor and cannot benefit from the same kind of thinking that goes into designing software. However, after spending many years involved in thinking about how to improve the process of software development, experience has convinced me that a great deal of the process of developing software can be treated as infrastructure. This infrastructure supports the creative parts of the software development process, but can be clearly delineated from it.
By thinking about the infrastructure of your process in the same way as you would think about any manual process that requires automation, you can leverage your skills as a developer and apply them in a new way. The process of developing software is, in effect, an algorithm implemented with various technologies. You can think of your team, the technologies you use, and your development methodology as software.
What technologies are you using? What is the architecture of your organization? Will it scale from its current size to double its size? Will it scale seamlessly to include new teams in new locations? What will happen if you acquire a company?
When creating software, you want to design it so that it is flexible and adapts to new circumstances. The same should be true for your development organization. In this article, I apply this way of thinking to the problem of scaling continuous integration to development efforts of more than 10 people.
In a typical project, the integration phase is towards the end and is the source of a great deal of problems and delays. One practice that has emerged to address this problem is continuous integration. The basic idea is that if getting build and test results on a regular basis is a good idea, getting build and test results on every change is even better.
With continuous integration, all work is integrated into a single codeline as frequently as possible. Every check-in automatically triggers a build and often a subsequent run of the test suite. This provides instant feedback about problems to developers and helps to keep the code base free of build and test failures. It also reduces the integration headaches just prior to release.
I'm a big fan of continuous integration. I've used it and recommended it for many years with great success. But it has a dark side as well. The larger and/or more complex the project, the higher the chance that it devolves into what I call "Continuous Noise." In this case, you get notified every 10 minutes or so (depending on how much building and testing is going on) that the build and/or test suite is still failing. It may be for a different reason every time, but it doesn't matter. It is tough to make progress when the mainline is unstable most of the time. This problem is not caused by CI, but it is exposed by CI.
The question is, what is a good way to structure this integration so that it will scale smoothly as you add more people to the equation? A good starting place is to look around for a pattern to follow. What are some similar situations? I have found that everything your organization needs to do in order to produce the best possible development organization can be entirely derived from the patterns and practices at the individual level. It makes it much easier to understand and much more likely that it will be successfully followed.