Checkpoints
In the face of all this potential chaos, it is comforting to have safety nets. If something goes wrong, you want to have an easy way to revert to a known good state. "Known good" doesn't mean "known to be perfect in every way," it just means a state that is both a well-known state and a state that has a quantifiable level of goodness. For instance, if you know your project built as of one hour ago with no warnings, then the previous state is a known good state. You know that if you revert to that state, then at least it will build again. Even if what is built won't pass all of your automated tests, at least it is a step in the right direction.
One of the best known forms of checkpoints are check-ins to private branches. By creating a private branch, you get all of the benefits of software configuration management (SCM) for your own personal use.
Check-in Early and Check-in Often
Everybody puts off checking-in their changes, sometimes for days. You put it off because you don't want to affect other people too early and you don't want to get blamed for breaking the build. But this leads to other problems such as losing work or not being able to go back to previous versions of your work. Sure, you can copy files aside in temporary directories, but then you are doing what an SCM system can do for you.
Once you reach a new known good state, create a checkpoint. In other words, do a check-in to your private branch. The checkpoint ensures that your work is safe and that you can return to the checkpoint at any time. Your changes are stored on the server and available to other users if they need them, but don't cause problems the way that making them public can. This frees you from worrying about losing progress due to mistakes or dead-end changes.
Development Hierarchy
A bigger picture solution to the chaos of mainline development is to use a development hierarchy. A development hierarchy is simply a hierarchical representation of the dependencies between groups that includes process steps such as integration, quality assurance, and code reviews. It is a natural extension of private branching. When you have a private branch, you have created a hierarchy. You check-in your changes on your branch first, then when you feel comfortable that your changes will have a positive impact and not break the build, you move them up the hierarchy by merging them into the mainline.
This can be extended recursively from there. Each team can think of itself as an individual that updates its private branch from the next step in the hierarchy and checks its changes back in via merging.
In Figure 1, there are four development teams as well as an area for accepting third-party code drops. Each of the teams are located in a different geographical area. Another level of the hierarchy that is not shown are the individual developers that are working in each team. The hierarchy represents the normal flow of changes through development from stage to stage. For instance, changes in the GUI team flow from individuals to the GUI stage to the UI_int stage, to QA, and finally to the release stage. Conversely, changes merged from the GUI stage to the UI_int stage are also applied to the Web stage and changes merged from the Web stage to the UI_int stage are also merged into the GUI stage.
Using a development hierarchy provides more opportunities for checkpointing. Every time changes flow through the system you can have a checkpoint. If it turns out that a change was undesirable, you can return both the source stage and the destination stage back to a previous checkpoint. By contrast, mainline development only offers you a single opportunity for checkpointing, the state of the main codeline itself. The chance of ascertaining the current state of the mainline is very low unless you prevent people from making changes to it for a period of time long enough to do a build, do a smoketest, or whatever level of validation you wish to do. Since the high level of churn in mainline development makes checkpointing so difficult, any available checkpoint is most likely very old and thus of limited value.
Another benefit of combining a development hierarchy with checkpointing is that each step up the hierarchy that a change makes adds another level of maturity to that change and each step up the hierarchy represents a higher level of maturity for the system as a whole.
