The Buzz About Builds

What's driving all the recent interest in Build tools that you might reasonably characterize as a rather, well, unglamorous?


June 08, 2007
URL:http://www.drdobbs.com/architecture-and-design/the-buzz-about-builds/199902659

At this year's SD West Conference, I was surprised by the number of companies (most of them new) focused on Build tools or tools with added Build functionality: Products like AutomatedQA's Automated Build Studio and Urbancode's AnthillPro3 Build Management Server and Build functionalities built into Nokia's Carbide.C++ environment. Today's Build tools are a far cry from make. Build is, against all logic, sexy.

So what's driving this increased interest in Build tools and the plethora of new products in a corner of software development that one might reasonably characterize as rather unglamorous? After all, we're just talking about the process of turning a bunch of interdependent chunks of already-written code into an application, an activity to which, when you're intently focused on actually writing the code, you'd probably like to devote as little thought—and as little time—as possible.

I asked that question of some experts in the field and did some research of my own, and the conclusion I was driven to is simply that it's Build's turn. There is not one reason for the current Build buzz, but several interrelated reasons, having to do with complexity, distributed development, diversity of development approaches, and compliance. And these forces are resulting in some powerful new prescriptions for easing the pain of the Build process.

Complexity

Anything you build on a large scale or with intense passion invites chaos.

—Francis Ford Coppola

Life happens too fast for you ever to think about it.

—Kurt Vonnegut

The growing complexity of software obviously creates a need for better development tools, and that includes Build tools. "Applications keep getting larger and larger," says John Ousterhout, the creator of Tcl and founder of Electric Cloud, a software production-management company, "with the result that Builds take longer and longer." He points particularly to the embedded device space, where Moore's Law increases in chip memory make possible larger and larger applications.

Factor in multiple variants of the software for different chip sets, and the Build process turns into a major time sink. "At one point," Ousterhout says, "LSI Logic had a software package that took two-and-a-half hours to build each variant, and they had to build about 30 variants." Slightly over three days for a full Build. Ouch.

But complexity impacts all aspects of software development; why so much focus on Build tools in particular?

Apparently, it's because it's Build's turn. Ousterhout sees Build and test tools as offering the greatest opportunity for improvement right now precisely because Build and other backend tasks have been largely ignored by commercial software companies until recently. Electric Cloud uses the term "software production" to refer to backend tasks such as Build, package, test, and deploy; the analogy is with movie production tasks that occur after the frontend activities create the raw footage. The frontend tasks in software development, like code editing, debugging, and source-code-control, have had the attention since the '80s and the major gains have been made there already. But the backend processes "are ripe for improvement." There is simply more room for gains to be made.

That's one factor.

Compliance

The triumph of anything is a matter of organization. If there are such things as angels, I hope that they are organized along the lines of the Mafia.

—Kurt Vonnegut

Another of the factors behind the need for more powerful Build systems is the hassle of compliance requirements, and related reporting and tracking concerns (see "Living with Compliance", www.ddj.com/dept/architect/188700752). Beyond the practical necessity of keeping track of where the pieces of a program came from, what went into this release, what packages are affected by a change, and what bugs were fixed, there may be governmental requirements to meet and reports to file.

"There is more and more pressure to make processes more repeatable, better documented, and more trackable," Ousterhout says. This demand for accountability requires better automation and reporting from the Build process.

Here, traditional Build scripts just don't cut it.

Tracy Ragan, COO of OpenMake, a Build-to-release management solution company, says, "With Build scripts, you cannot match what source code was used to create the binary. Even when a bill-of-material report is produced during the Build, this is simply a listing of what was managed by the version-control tool, not what went into the Build."

And when dealing with audits, you need more than this. "We have seen audits fail," Ragan says, "because after some investigation it is exposed that the versions of SOA objects and Java runtimes were not reported in the bill-of-material report, and the developer's best guess did not match production. End result: Production breaks and audit fails."

So that's another factor driving the buzz around Build.

Distributed Development

We Bokonists believe that humanity is organized into teams, teams that do God's Will without ever discovering what they are doing.

—Kurt Vonnegut

Then there's time-zone-shifted programming: Offshoring, outsourcing, nearshoring, open-source collaboration. Distributed development, in other words. As explored here in the June 2007 issue with regard to software development in Eastern Europe, it is increasingly common to have programming teams whose members work in different time zones. Motorola, Symantec, Qualcomm, and Freescale are just four examples of companies with labs in multiple countries, doing software development across time zones.

This can work well, apparently, at least sometimes, for getting the coding done. But it is likely to wreak havoc with the Build process. "You can't have a single central Build-meister fixing all of the problems," Ousterhout says, "when half of the team is in India and half is in Silicon Valley. The Silicon Valley Build-meister isn't even awake when something breaks in India."

The more distributed the development process is, the more need there is for automation of the Build process. "In general," Ousterhout says, "you need more automation for the interactions between teams when they don't overlap in their work shifts."

So the increasingly globally distributed nature of software development is also a factor.

Just Build It

Everybody wants to build and nobody wants to do maintenance.

—Kurt Vonnegut

Developers not only set their clocks differently, they may code in different languages. A decent-sized software project is likely to involve several languages—in addition to the C or C++ or Java code, maybe a smattering of Python or Tcl or Perl, and some XML. Even an individual programmer working alone is likely to work in multiple languages/scripting systems. The good news is that there are well-established Build tools for C, Java, Perl, Python, Ruby...

The bad news is that there are well-established Build tools for C, Java, Perl, Python, Ruby...

Ragan: "If you are writing both Java and C apps, then you have to write separate scripts, written in different Build scripting languages (make and ant) to get the job done. What a headache. We get customers asking, 'how can I create a single process for both languages' all of the time."

That's not a trivial challenge. "Java works differently from the traditional C compilers," Ragan says. "This causes some really convoluted Java scripts. As a person who can easily read a make script and is versed in XML, I struggle following the logic of the Java scripting process. There is no clean way of doing it, I don't care how good a scripter you are."

Then there are the demands of Agile development. Mario Moreira of CM Journal has predicted that one of the hot issues for configuration management this year will be Build automation agility. He points to IBM's Rational BuildForge Enterprise edition, open-source Luntbuild, and CruiseControl, and Urbancode's AnthillPro3 Build Management Server as tools providing the continuous-Build requirement of Agile development. He could also have mentioned OpenMake, JetBrains TeamCity, Zutubi Pulse, PMEase QuickBuild (based on Luntbuild), Electric Cloud's ElectricAccelerator/ElectricCommander, VSoft FinalBuilder, ViewTier Systems' ParaBuild...

"One of the key principles of agile development," Ousterhout says, "is frequent integration and testing, both of which depend on reliable and efficient backend processes. Many organizations are trying to become more agile, but discover that they cannot do it without major improvements in their backend processes."

Why? Because you can't do Agile with long Builds.

"If you are an agile developer," Ragan says, "you want to do Builds continuously. Well, if you're a Java developer using ant or maven scripts, then you cannot do an incremental Build. Incremental means if only one file changes, just rebuild what was impacted. So instead you rebuild all. Well, a rebuild all could take hours, meaning continuous integration waits for the last Build to complete."

Ragan's and Ousterhout's companies exist to tackle such challenges. Ousterhout points out that Intuit uses his company's ElectricAccelerator tool to rebuild QuickBooks automatically "every 30 minutes all day long, every workday." That way, overnight Builds rarely fail.

Another factor or two.

The Building Boom

History is merely a list of surprises. It can only prepare us to be surprised yet again.

—Kurt Vonnegut

Of course, it's not as though nobody was working on Build tools and systems until the last year or so. In fact, there was a lot of development going on: Jim Weirich's Rake, to cite one example. An ant-like Build language written in Ruby. (Martin Fowler has pointed out the perhaps obvious but still interesting fact that Build tools such as ant and Rake are in fact languages, but languages that use a different computational model than the developer probably is using in the code they Build: dependency-based programming. Thus, you are programming when you use these tools, and in a less familiar paradigm.)

But for the most part, these tools have been elaborations on make, which has served the UNIX world for decades, plus ant in the Java world, and in the .NET world, what—Nant and MSbuild? In general, Build tools have not been venture capital-attracting commercial endeavors. And as recently as a year ago, it was possible, in the right setting, to say, "Everyone uses ant," without being contradicted ("Tips for J2EE Development," February 24, 2006, AWProfessional.com).

Now, though, a lot of companies have seen dollar signs in the Build market. And there will be more: "Success feeds on itself," Ousterhout says. "For years, software development managers resigned themselves to the idea that Build problems were just...a limp we can live with'." But when serious Build tools like Ousterhout's and BuildForge pointed the way, there were two effects. "Development teams are starting to realize that this is a limp they don't have to live with [and] entrepreneurs are seeing the commercial success of the leaders." Selling your young company to IBM, as BuildForge did last year, is one such sign of success.

Which means that even more companies will enter the Build tools market, creating new, innovative products.

So it goes.

—Kurt Vonnegut

Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.