Here are a few common areas where companies seem to have the biggest struggles with regard to implementing effective continuous delivery strategies.
Companies often find themselves struggling to define small deliverable stories that are easily understood, managed, and delivered. The solution to this problem is to set goals to make each story smaller, and then focus on continually improving the story writing process until stories can be finished with very short cycle times. Feature toggles can be used to hide stories until a complete set of related stories are completed, or to allow A/B testing of the usability of a set of stories.
Focusing on the Wrong Tests
The state of software development testing today sees most organizations investing heavily in GUI testing. This practice is actually an impediment. Organizations should put the majority of test focus on unit testing, where test coverage is at its greatest. From there, focus should next be placed on integration testing, in which a small number of integration tests are used to ensure all the pieces work together. GUI testing should have the least amount of focus because GUI tests are often brittle, slow, unreliable, and the most expensive to maintain for the value they deliver. The result of this testing strategy will yield faster cycle time, higher quality, and lower test maintenance costs.
Scaling Continuous Integration
Another frequent problem is scaling continuous integration. As development scales, you get many more concurrent changes in progress, builds take longer and tests take longer, resulting in broken builds. In the worst case, builds on a single mainline become mostly broken instead of mostly working. That is not continuous delivery. Worse yet, the unstable main line affects all developers and slows down their productively dramatically.
There are two primary techniques companies employ to achieve continuous delivery at scale. Each of these techniques is designed as a "divide and conquer" strategy, allowing for focus on specific areas which can have significant impact on continuous delivery:
- Component or Service Architecture: Companies can choose to change their software to a component or service architecture. These are components that can be individually deployed in production independently of other components. They have clean interfaces and are compatible across multiple releases of components. This structure provides scalability because you can then run separate development and delivery pipelines that don't interfere with each other.
- Multi-Team Continuous Integration: In this approach, scalability is achieved by doing development using separate teams. You then join the work from those teams as they go through integration and test processes, and eventually, out to the deployment pipeline. A key to multi-stage continuous integration is to break down the development pipeline and integration testing into stages. Organizations may define these stages by team branch, feature branch, build (nightly, weekly, etc.), and main line. Using integration testing along each of these stages provides a level of test coverage and continuity that allows code to flow up the pipeline while also identifying issues faster and closer to the developer, where the cost to fix the issue is less expensive.
An example of successful scaling continuous delivery I have seen is with an international online gaming company with 70 developers organized into 10 teams. One of the keys to their success was the change management process they have incorporated into their continuous integration practices. Due to the nature of the gaming business, inter-product dependencies cause a rate of change per product on a daily basis that is sometimes as high as 60%. Using effective continuous integration and change management techniques, this company was able to handle this rate of change, and as a result, is producing more than 3500 releasable builds per month. This company fully embodies the continuous delivery principle of always having releasable code.
Using continuous integration and continuous delivery effectively can dramatically improve the way your engineering team operates, as well as the business overall. Companies following these best practices will gain significant competitive advantages in software development time-to-market, agility, and quality. The key to being successful with continuous delivery is to start now. The faster your organization can begin avoiding the pitfalls of lengthy development cycles where engineering teams are stifled by manual processes, the faster it will realize the benefits of continuous delivery.
Dave Jabs is the CTO of AccuRev and a strong proponent of continuous delivery.