Building a Robust Development Environment

Stephen presents strategies for source-code managment, build automation, and human factors—including tips on how to handle significant project change, regardless of platform.


September 03, 2008
URL:http://www.drdobbs.com/tools/building-a-robust-development-environmen/210300282

Stephen is a software engineer at CDW focusing on web scalability and best practices. He is also a founder of the Chicago Architects Group and can be reached at [email protected].


In the southwest United States, Native Americans refer to the combination of corn, beans, and squash as the "Three Sisters." The crops were grown in close proximity because they complemented each other and balanced the nutrients in the soil. This way all three crops could be planted year after year without rotation. Software engineering has its own Three Sisters—source-code management, build automation, and human factors—which are as important to the engineering process as nutrients are to soil, and provide the basis for repeatedly shipping excellent software.

Gaining insight into these aspects of the engineering process can be a make or break situation for a team. The Three Sisters take time and effort, but yield big results in quality and developer productivity. Having recently led an effort to reconstruct and revitalize the software configuration management process within my company (for our e-commerce website), I can attest to the value that is attainable at many levels of the software engineering process.

In this article, I present strategies and insights for source-code management, build automation, and human factors—including tips on how to handle significant project change, regardless of platform. You won't find the theoretical here. Instead, I present real-world pragmatic practices that have worked.

Finding the Source

Taking charge of your source code is integral to software engineering. Before I started the design activities for the new build system, I addressed two glaring issues—an antiquated source-control system and a confusing source-code directory structure.

All source code lived inside of Microsoft's Visual SourceSafe, which didn't provide any benefits beyond a library-like file check-in/check-out system. Replacing this software became a priority and Microsoft's Team Foundation Server (TFS) was brought in as a replacement. Although there are many good available source-control systems—Subversion, Perforce, and SurroundSCM come to mind—TFS provides seamless integration with the Visual Studio.NET IDE on every developer desktop. The introduction of TFS provided a new and improved platform for development and build engineering to expand upon.

Next, the source structure needed refactoring—the website code was mingled with other company code, using namespaces as a folder naming convention. Even though namespace-based folder structuring can sound appealing to .NET and Java projects, it can get out of control and lead to nesting problems. This is not always the case, as namespaces in either language are meant to span physicality. But without being very careful and having a core team responsible for the structure, complexity can sneak in. There's usually enough complexity in the actual application design that it's wise to keep the physical source structure as simple as possible.

A flat-folder structure applied itself well to resolve the folder structuring problem. Each project composing the greater website solution received its own renamed project folder and lived adjacent to the other projects, eliminating all project nesting. Now project groupings happen at a logical level instead of the physical folder level, making the structure less brittle; see Figure 1.

Figure 1: Project groupings.

Perfect Source Structure?

Is there a direct connection between how source is structured and how software is designed? No, not in compiled software. When software compiles, this is an indication of its abstraction from the source files. Don't confuse application architecture with source-code management. It's possible to achieve your design goals and reuse across products without heading down the path of sharing source code. The usual suspects of binary compatibility, RPC, and web services are answers to this question that do not effect source code nor introduce additional complexity.

Don't make the mistake of trying to create the ultimate source-code repository as a companion to an idealized application architecture. If the history of software development provides any guidance, then one of these is probably a chimera. Ownership of source code is important for its health and maintenance, therefore some group must take ownership. To accomplish this, ensure that all source code is accounted for and part of a team's roadmap. Avoid getting caught in unrealistic expectations of architecture driven by the source management process. There isn't a singular application architecture that must be subscribed to. After all, every system is unique and requires tradeoffs for longevity. To move the process forward, it's essential to help stakeholders understand that there are many scenarios—some better than others—for structuring source. Strive for as good as possible for the reality of your teams.

Branching and Continuous Integration

Continuous integration (CI) was the initial—and still long-term—goal for build automation and concurrent development. But things aren't always that simple. Re-engineering a source-control tool or folder structure are child's play compared with changing how projects are planned and executed. (Read Scott Ambler's "The Agile Edge" column if you don't agree.) A goal of CI is to minimize the number of branches that are necessary for development isolation. This is accomplished by breaking down bigger projects into stories, which can be planned and executed on the trunk of the source tree. For better or worse, this kind of large-scale change in project planning is difficult to attain without direct input or the influence of project planners. In my case, project planning was happening at the same time as new build engineering activities. Synchronizing these was almost impossible. If your organization is one that depends on detailed functional specification documents and you don't see that changing anytime soon, you'll need to be practical and compromise.

With pure CI no longer a realistic option for allowing concurrent development, different branching strategies were investigated. There is plenty of guidance out there on branching, but most of it can be distilled from the smoke and mirrors into one maxim—"branch to isolate." When you set a goal to allow isolation, the details around how and when to branch become much simpler. For example, part of the reason we couldn't roll out a more authentic form of CI right away is that projects are in the pipeline up to a year out, but don't all occur serially. They overlap, touch, depend on one another, and consequently, require coordination. Allowing this isolation keeps development rolling as opposed to deteriorating in fire fights on what CI or agile development is and how to get there; see Figure 2. Don't be legalistic: Trade continuous integration for continuous improvement when necessary. Being flexible can bring only benefit in the end, while being rigid in opinions or actions causes your attempts at change to fracture.

Figure 2: Project branches.

That said, some CI concepts are still applicable. Using a tool like CruiseControl.NET allows build automation per developer check-in. There is still immediate developer feedback on the status of the build, except it's not just one build—it's for all isolation branches. It is critical to use a tool (like CCNET or another) to automate CI or you'll soon be spinning your wheels on the minutiae. Figure 3 illustrates multiple branches in isolation.

[Click image to view at full size]

Figure 3: Multiple branches in isolation.

Under this type of setup, the build server turns into your development lifeline. Its activities are viewed with glee by the process-oriented on the team, or derision by the cowboys. No matter the opinion, this server is extremely important as the workhorse of the team. Making the build fast, concise, and as useful as possible changes opinions dramatically. The CCNET server deployed here used NANT scripts to generate a unique version number embedded in each compiled component. The version number is also used to label the source code after a build and provides traceability of the component all the way to production. After the compile and other tasks, the artifacts are deployed to a predetermined environment. This way, the process is automated, repeatable, and consistent. Take advantage of this kind of infrastructure to add value beyond a compile and deploy.

You should also follow a deployment methodology where branches are not candidates for release. Any given branch must be merged back into the Trunk for integration, be approved, and then have an additional release branch created for it. This creates an additional level of isolation—this time between your testing environment and your development environment. Following these strategies puts you in a good place to move toward true CI if and when the time occurs. Always think long term with these decisions; they form the bedrock for development activities.

Automate

Again, the build process is more than compiling source code. There are other activities that need to occur to test and use the newly built software. These activities are usually domain specific—perhaps triggering a series of unit tests or starting a system stress test. In my case, a web environment with Microsoft IIS 6 is created for each branch. The ability to quickly build out an environment for a developer or testing server can free precious cycles from your team. Spending time creating a tool to accomplish this yielded big results. The tool can setup the IIS website on any server identical to production. Then, spinning up a new environment is as simple and repeatable as compiling the software that executes in the environment. Are there items in your test environment that are difficult to reproduce? Are developers ever held up changing configurations? Look for opportunities like these to automate as much as possible.

Delegate Responsibility

Branching is considered an administrative task and should only be done with team or management agreement. Small checks and balances here prevent bloating of branches and give the development team a signpost to follow. Once the branch is created, someone must be responsible for it. Project leads fit this role best; they can merge changes from the trunk into their branch on a regular basis, or delegate the task to another member on their team. Delegating these responsibilities helps the development team feel ownership over the source code and process.

Handling Change In the Platform

Changes in your core platform are inevitable over time, but they don't have to be "harbingers of doom." Making the change a success, for the team and your build process, comes down to how you decide to handle the change. For instance, you may decide to pass up a couple of point releases of the .NET Framework (or Java JDK) to avoid instability or new engineering hours in the build process. Gauge how this sits with other stakeholders. Are they disappointed? Does skipping this upgrade hold up their development roadmap? Are they angry? These are activities that can't be settled easily via e-mail and require face-to-face interaction. Be proactive about gathering these details because the effect can be significant.

A platform upgrade in my environment from the .NET 1.1 Framework to the .NET 3.5 Framework was planned over a number of related product releases, with the first release still on .NET 1.1. This can feel like shell shock to those responsible for the build and source management. Two versions of the source must be maintained for a period of time without interfering with each other and be built with different compilers.

To accomplish this, create a new branch from your source code's trunk for the first release. Then upgrade the trunk source code to the targeted platform version. You can now place one team on the old version and another team on the upgraded version. Once the release on the old version is complete, you can reverse merge those changes back into the trunk (see Figure 4). Be careful not to perform the platform upgrade on a development branch; that is, an open invitation to branch on a branch, which gets unruly quickly and is always best avoided. The branch -> upgrade -> release -> merge process is a prime example of the branch isolation advantage in the wild. It also provides a baseline of the code before the upgrade, allowing an easily identifiable rollback path.

Making similar platform accommodations garners support and respect for the build process.

[Click image to view at full size]

Figure 4: Reverse merge.

Human Factors: Managing Fear

Underemphasizing the social challenges of software engineering can sound appealing but can produce additional work, or failure, in the future. The changes described here are always possible—there are simply consequences to deal with. The most prominent one that I have experienced is fear.

The business stakeholder fears a development slow down if the source-code repository is changed and a build process is installed. These folks have goals they are trying to achieve via the development teams' output. Software configuration management, build engineering especially, puts these goals at risk—or at least appears to. Your job is to point out the great advantages of the new processes, so education is key. Explaining exactly what the branching strategy is and how concurrent development will make more releases of a higher quality possible helps change the attitude towards the work.

Development management may seem like an easy win, but that's not always the case. If your team hasn't had a structured build and release process in the past (and your systems are functional), the need for the large investment of time and resources may not make sense to them. The education here is different than for business stakeholders. Present these advantages to your team:

These are all big wins and make management look good.

The last, and toughest, audience that needs education are software developers themselves. If development management seemed like one that should be easy, the development team can be even more averse to change. This is because your everyday developer is not simply talking about projects and ship dates. They are hacking the software out line-by-line, feature-by-feature. Causing big waves here can sink your efforts before their next release. Fortunately, this is overcome by helping individual developers change their habits. If you have a mid-size to large team, choose one or two developers to represent the new process. Teach them how to work within the new branches, environments, and the continuous integration dashboard before you educate the rest of the team. This way, you have some advocates on the inside.

Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.