Dr. Dobb's | Separating Duties to Meet IT Compliance

Separating Duties to Meet IT Compliance

If your organization is facing strict audit compliance but struggles with distributed platforms, take a look at how the big boys do it.

August 02, 2007
URL:http://www.drdobbs.com/tools/separating-duties-to-meet-it-compliance/201202677

Tracy is chief operating officer at OpenMake Software. She can be contacted at [email protected].

We used to call it "throwing it over the wall"—the process of turning over source code for production release. That was in my mainframe days when "throwing it over the wall" was a simple check-in of source code to CA Endeavor. Once we handed over our source code, the rest was handled by production control. The request for approval to promote to the preproduction stage was all that was required. Production control handled the promote, compile-and-link edit, and move-to-production. This simple process of separating production control duties between development teams and production control teams has been a standard process on the mainframe for the last two decades. But developers of today's distributed platforms—Java and Windows developers—just can't seem to figure it out.

But is the distributed platform really that complex, or are we missing something? To answer this, look at large UNIX-based legacy systems. Written in native languages such as C or even assembler, they resemble both the Java/Windows and mainframe worlds. Ask any systems administrator on the UNIX platform. They have the bragging rights to describe a process that clearly separates duties between development and production control. Developers check-in their source code to IBM ClearCase and the system administrator manages one large ClearMake makefile to build the code and move it to production. Different tools, but a similar process to the mainframe. If complex legacy UNIX application teams can do this, then so should Java and Windows application teams.

These legacy processes should not be forgotten. They have much to offer in terms of lessons learned from years of experience delivering critical software solutions to end users. They represent the most sophisticated systems ever designed. They serve as the intelligence in our Apache Helicopters and track our hard-earned cash through complex banking systems. There is nothing simple at all about the applications, or the code that these systems deliver.

Common Themes

There is a common theme in both of these "over the wall" processes—both hold true to a very clear separation of duties between development and production control. The key to the process is that developers never build the production executables, nor do they control the process or scripts that create the production executables—and they never release the production executables to production. In both of these environments (development and production control), binaries can be traced back to their source code of origin through footprint technology that captures the information at compile time. Developers are responsible for developing and unit testing the source code, then leaving the rest to teams that focus on getting the steps right.

Java and Windows development teams are now being asked to step up to the plate and hit one out to the cheap seats. Auditors and government regulations such as Sarbanes-Oxley are simply expecting from Java and Windows developers the same level of professionalism that has been demonstrated by developers who came before them. In many shops I visit, I hear CTOs and CIOs complaining that they could pass their audits if only they could put some controls around the Java and Windows development teams. They can't understand why, after spending money on versioning and sophisticated software configuration management (SCM) tools, that they still cannot provide a clear audit trail showing what source code was used to create the binaries running in production—and why they cannot restrict Java and Window developers from having access to production servers.

The answer is not too far in the past. CIOs and CTOs simply need to look at their legacy mainframe or UNIX applications to see the answer. Java and Windows developers struggle with letting go of the process of building and releasing their binaries in the same way as mainframe or UNIX developers can. Most importantly, Java and Windows developers have not sorted out a build process that can be repeated regardless of build location. Their build scripts are ad hoc, have static references, are highly redundant, and are anything but repeatable. For this reason, they "hit the wall" instead of going "over the wall."

Binary Management

The application build process is the final piece of the IT compliance puzzle for distributed teams writing Java and Windows applications. Ad hoc build scripts are the roadblock preventing Java and Windows developers from moving forward. Ad hoc build scripts are the most common methods used to manage Java and Windows application builds, yet they don't meet the four essential requirements of IT governance standards:

Traceability
Auditability
Validation
Separation of workflow duties

To mature the Java and Windows development process, a method for managing binaries must be implemented. Managing binaries is a core component of the build-to-release process. If you cannot manage the process of creating binaries, then your build-to-release process becomes invalid. Similar to SCM, binary management lets you track, trace, and manage the details about the build. Your process of managing binaries lets you control what compile-and-link flags were used to build the deployable objects, restrict the use of debug flags used in production builds, and control the location and version of the compiler and linker used in the build. A wobble in any of these areas can cause drastic differences in build results.

There are four core competencies to a solid binary management process:

Reduce your dependency on ad hoc build scripts. Both mainframe and legacy UNIX developers had to face this core requirement. At one time, mainframe developers managed their own compile JCL. They would release applications into production by running their compile JCL and creating load libraries right into production. Legacy UNIX developers wrote hundreds of make scripts until they realized that a single well-managed makefile written in a commercial make language such as IBM ClearMake offered them a method of standardizing the creation of binaries across diverse teams. Java developers are overly dependent on open-source languages to perform this critical step. Windows developers often rely on point-and-click processes inside Visual Studio to create the binaries. Neither method allows for a repeatable process that can be turned over to a production control team. As long as you rely on ad hoc build scripts, you will never meet the separation-of-duties requirement.

Never take binaries from a developer or execute a developer's build scripts. The only reason production control would rely on developers to create binaries or to provide scripts to create the binaries is because there is no repeatable method between development and production. Therefore, all of production control must be done in the development environment. The requirement to never take binaries from a developer is clear, but understanding why production control should not execute the developer scripts is less obvious.
If your production control team is simply executing an automated process to run a build script provided by the developer, the separation-of-duties requirement cannot be met. First, there is no clear understanding where the source code is coming from. Scripts point to directories that are often outside the local build directory. Second, the developer's build-machine configuration may be at higher versions than the production runtime environment. And finally, your production-control team needs the ability to validate if production-level build configurations are turned on/off, such as removing debug flags and adding optimization flags. Yes, it is these steps that mainframe and legacy UNIX applications have understood for years, but Java and Windows teams are completely forgetting.

Identify the files used in the build that were not managed in the SCM repository. If you have gone through the expense of implementing source-code management tools throughout your development process, then it is critical to validate that the managed source code is actually used in the build. It is often critical to identify the components used in the build that are not under SCM control. These components are often critical third-party libraries, database libraries, SOA WSDL objects, and Java runtime libraries. Your process of managing binaries cannot underestimate the importance of these components. They serve to be as critical as any homegrown source module.

Create a footprint showing which source code and libraries were used to create the final executables.

Managing Dependencies

Managing dependencies is the most critical requirement of binary management. If your build-to-release process does not manage dependencies, then you won't be able to clearly see what components were used to create the executables. Yes, you can generate traditional bill-of-material reports that show the files checked out of your SCM tool, but this does not expose what files were found outside of the local build directory. Dependency management and orchestration provides a complete audit trail showing what source code and versions were used to create the final deployable objects. Nothing else can do this. Dependencies can be difficult to trace and often impossible to understand with manual scripts. That's why you need to implement a build-management solution that offers binary-management services to ensure that when the build executes, a dependency-scanning tool watches exactly what is called and used by the compilers and linkers. This is familiar to the mainframe of legacy UNIX teams. Figure 1 is an example footprint that exposes not only the files managed within IBM ClearCase, but also the files not under ClearCase control; notice rt.jar is listed as "Not in VOB." This footprint also exposes all of the environment variables used during the build. This level of footprint goes far beyond what a simple bill-of-material report can show.

[Click image to view at full size]

Figure 1: Example footprint that exposes files not managed within IBM ClearCase.

If you're a Windows or Java developer and think that the use of ad hoc scripts is the only way to complete the job, well, I understand. I thought the same thing years ago when I was told I could not use my own compile JCL to release my mainframe changes to production. But there are real benefits if you implement a build system that actually managed your binaries:

Developers using continuous integration builds could benefit from performing truly iterative (incremental) builds. If you have a need for speed, the easiest way to get there is to stop building objects that are already up-to-date. The best way to accomplish a continuous integration build process is to build only the changes each time a build is launched. The concept of continuous integration is not new. Just go ask one of the mainframe old timers. They've been building in an iterative, continuous method for years. They do it best—they check-out, make a change, and check-in. The check-in launches a build that only builds what has changed. They never need to perform a "clean all" build (and would laugh at the thought). If they did, some large systems could take days to compile. This point alone should pique your interest.

Simplify changes outside your IDE as easy as inside your IDE. I see Java developers struggling with code and package refactoring outside the IDE all the time. It's incongruent that developers demand their IDEs handle processes such as code refactoring inside the IDE by pushing a button, but when building outside the IDE they are required to manually revisit dozens of Ant scripts to reflect the refactoring changes performed automatically by the IDE. Isn't it time for your build system to be dependent upon the IDE project file, preventing the manual update of Ant scripts anytime refactoring is required? There are better ways.

Binary footprints expose the problem areas in a fast and efficient way. Distributed developers (Java or Windows) could benefit from what the legacy UNIX and mainframe teams have used as a critical tool for years—the ability to look inside the binary by running an "identification" program that shows the precise artifacts used to create the binary. There is no faster way to identify which source code or library broke the build or caused a runtime failure.

Conclusion

If your organization is attempting to meet strict audit compliance but struggles with meeting the separation-of-duties requirement when it comes to the distributed platforms, consider taking a look at how the big boys do it. Legacy UNIX and mainframe developers went through the same growing pains that distributed platforms are currently experiencing. Meeting that separation-of-duties requirement is as simple as implementing a build-to-release management system that supports binary management, and letting someone other than a few core developers build the application on only one or two development machines.

The benefits of addressing the binary management step of the build-to-release process will pay off in both time and money as well as the maturation of the Java and Windows development process. And don't be concerned that meeting this requirement somehow interrupts your lean development techniques—it can actually improve them. Both production control and developers have lots to gain by solving this critical component of the development-to-release process.