Intuit Inc. was founded in 1983 as the publisher of Quicken, an MS-DOS financial-management system for individuals. Quicken's success led to a host of follow-up products, including QuickBooks, which was launched in 1992 and quickly became the industry's top small-business accounting solution. Intuit describes QuickBooks as the backbone of its "small business ecosystem," a market that accounts for a major share of the company's $3.9 billion in annual revenues. It is easily the best-selling retail software for small-business accounting worldwide.
Dr. Dobb's spoke with Jon Burt, group manager in the operations department for Intuit's Financial Management Solutions division. FMS supports the delivery of QuickBooks for Windows, QuickBooks for Macintosh, and QuickBooks Online. Burt manages software configuration management for these three products, plus the operations aspect of QuickBooks Online. He manages a staff that is split between Intuit's Mountain View, California, headquarters and Bangalore, India.
In addition to supporting multiple platforms, Intuit offers different feature sets at different price points. QuickBooks for Windows, for example, is available in Pro, Premier, and Enterprise editions. Specialized editions are available for accounting, contracting, manufacturing and wholesale, nonprofits, professional services, and retail businesses. Additionally, each product may be purchased on CD or via subscription. Subscribers get new features as they are developed, while CD users don't get the new features until they upgrade to a new release of the software.
As if that weren't enough, the software is also offered in localized versions for the U.S., Canada, and the United Kingdom.
A Single Codebase
"You wouldn't guess it," Burt says, "but on each platform, all those products come out of a single codebase. The Windows version is about 80,000 source files, 10+ million lines of C++ code plus a little C# for the .NET parts. Plus help files, tax tables, files defining local accounting rules, tax and other government reporting forms, upgrade offers…a lot of files. Every customer gets the full version. Specific feature sets are turned on and off with the license key."
The license-key approach means Intuit can upgrade customers or license specialized versions of the software with a simple online transaction. It also has big implications for SCM. "The benefit is that we're always extending and refining a single codebase," Burt notes. "We don't have to worry so much about maintaining compatibility among multiple versions of the product. The downside is that a full build rolls out many, many versions of the product. It's a huge build."
As the manager with primary responsibility for build management, Burt has supervised the construction of automated systems that perform continuous builds and continuous integration. The systems incorporate tools for testing, version control, and scheduling.
The decision to use a single codebase for each platform was a strategic one for Intuit. "It's a lot of work," Burt acknowledges. "It means that we needed to put coverage tools in place so we could keep complexity under control as we added features to the product. We needed to perform frequent builds, so we needed to invest in systems and tools to turn the codebase over as quickly as possible."
Managing Multiple Builds
In addition to different combinations of locales, versions, and vertical-market features, Burt specifies that each build includes a debug version and a release version for each locale, a version built for analysis with Coverity, and a build that incorporates HP's Fortify code-analysis tools for security scanning. Plus, Burt says, there are separate builds for the main trunk of the source-code tree and for branches associated with different projects about four branches per year, on average, each with a full set of builds. That's a lot of specialized builds, and it requires a lot of resources.
"The Coverity build alone is a 20-hour run," Burt says. "Coverity does a tremendous amount of analysis, so we threw major resources at it."
Intuit uses Jenkins, the open source continuous-integration tool derived from Oracle's Hudson, to coordinate builds. "We use Jenkins essentially as a job system," Burt says. "It's a scheduling app. A build is triggered when code is checked-in, or if there's a series of check-ins within a 15-minute window. Every 15 minutes, we ask the version-control system if anything has happened along each branch. If the answer is 'Yes,' then we have a set of continuous-integration systems that will go off and do a build."
The continuous builds are deployed to specially prepared virtual PCs (implemented with VMware) where they undergo minimal testing. "We do just enough testing to ensure you can open the application, create an invoice, and close the application," Burt says. "No one does anything with these builds they're disposable. They're basically to ensure that you didn't forget a semicolon that could break tomorrow's build. They're compiled with zero warnings. If you've been doing your local compiles in debug mode, you may miss or ignore a warning that's what trips people up. It's not reasonable to ask everyone to do his local builds in both debug and release modes, so we let the continuous-integration system do it. If the test exposes an error, everyone who had a check-in since the last clean run gets an e-mail, and they sort it out. You also get an e-mail when the tests succeed, so you have some assurance that you're not going to have a DOA build in the morning."
This minimal testing has a major impact on workflow, Burt says. "It's really the problem I came to Intuit to solve. Every morning we'd come in to a broken build. We'd spend from 9 a.m. to 1 p.m. unblocking QA so they could test. We were always behind."
Burt relies on Perforce Software's Perforce revision-control system, with good results: "Perforce works well for us. It's rock-solid, it has the features we need, and it's pretty fast. The main server is here in Mountain View, and we have proxies at the remote sites."
In addition to the check-in-by-check-in builds, Burt's team schedules nightly builds of the entire codebase, plus additional runs. There's a Fortify build every night, for instance. A weekly schedule governs which branches get Coverity builds on which nights.
Five years ago, builds were performed on a cluster consisting of 70 rack-mounted 1U nodes. Last year, Burt says, Intuit replaced that hardware with four Dell 2950 2U nodes. "We've got twice the performance now, after moving from two full racks to just four 2U units," he says. To take full advantage of the hardware, Intuit uses Electric Cloud's ElectricAccelerator to distribute builds across multiple virtual machines. ElectricAccelerator lets Burt's team launch compiles in parallel across the cluster. "We can build the entire product 8 to 10 million lines of code in 45 minutes," Burt says. "With ElectricAccelerator, we can do 7 builds in parallel and get 45- to 60-minute turnaround time. Of course, it's a diminishing-returns dynamic. The happy point is about 10 virtual boxes. If you do more, you hit the asymptote it's not worth it 10 seems the sweet spot for this codebase."
Burt says Intuit has been shipping product built in parallel for the past four years. "I'm not sure management knows we're not building sequentially," he says with a laugh. "But there's no reason for them to worry. It's solid. If we have a broken build, it's never because of Accelerator."