Channels ▼


Building QuickBooks: How Intuit Manages 10 Million Lines of Code

Automating QA

The end result of the build is a set of CD-ROM packages and downloadable packages for the Web. At that point, a custom-built app called AutoLab takes control. AutoLab sets up virtual machines and installs the various releases. First it preps the machines across multiple versions of Windows. Then it runs the installation, opens a company, and closes a company. That confirms that the build resulted in code that installs and runs. That's the basic acceptance test.

Once the code is loaded, the team runs feature scenarios and regression tests. For example, they may exercise all the options of the invoice system. This is all done with Micro Focus SilkTest. Intuit relies upon an outside team to maintain these tests. "You don't put your top talent on maintaining test scripts," Burt notes, "but you must have someone who is responsible for maintaining them. That's a natural thing to outsource."

The results of functional testing are delivered to developers every morning. Assuming the code works properly, developers sync their systems to the new codebase.

Unit testing isn't a big part of QuickBooks for Windows — the bulk of the codebase was written before unit testing was acknowledged as a best practice. QuickBooks Online, which is written in Java, incorporates more unit tests. In fact, Burt says, unit tests dominate the build cycle.

Burt is devoting significant resources to code quality this year. "We've sent multiple teams of five developers into the Coverity logs," he says. "They track down the warnings and fix hundreds and hundreds of minor coding issues. We're cleaning up little things that have been there forever. The goal is to ship a release with zero defects." This isn't just a matter of tools, Burt acknowledges. Team and management support are essential because they must remain committed to QA every day. "It doesn't matter what tools you use," Burt says. "If your team doesn't support it, if they're focused on adding features, you're wasting your time. If you don't want to use a tool like Coverity to track down defects, that's fine — don't use it. But if you say you want it, then you've got to commit. The payoff in code quality is tremendous."

Developers can be resistant to detailed analysis of shipping code at first, Burt says. But Coverity can surprise even senior architects sometimes. "They'll say, 'That's not a bug,'" Burt says. "And then they're like, 'Oh…I didn't think about that path.'"

Burt has worked with the development group to ensure that neither stigma nor penalties apply to developers who find code bugs during detailed analysis. "No one tracks the number of bugs attributed to you," he says. "That's something Intuit has never done. You break the build enough times — well, that gets people's attention. But not bugs. We're focused on eliminating defects, not apportioning blame."

Bug-tracking is handled with Serena Software's TeamTrack. "It's really overkill," Burt observes. "Most of Intuit's other teams have moved to lighter-weight tools, and we may do the same at some point."

Lessons Learned

Dealing with such a large codebase has been an education, Burt acknowledges. "The size of the codebase has spawned a' 10 Commandments of Things Thou Shalt Do Before Checking in Code,'" he says. "It's common sense stuff, but it's important. You're going to do a local build. You're going to run positive and negative tests. You're going to find someone to buddy-check it. If it's a bug fix, the comments are going to refer to the bug. Stuff like that."

Branching, Burt says, is a particular danger. "We have gone branch-crazy in the past," he says. "Everybody wanted his own branch for specific projects. That's great until you want to integrate the various branches back into the main system. If you wait too long and try to roll up all at once, you are in for an ugly month." In the past, Burt says, when a build took 4 hours instead of 45 minutes, it took a half-day to see if a branch broke the code. ElectricAccelerator was a game-changer in that regard. But still, Burt says, more than 5 or 6 branches is probably too many.

One surprising lesson is that small teams work, even for very large codebases — especially, Burt says, in sustaining an entrepreneurial, creative culture. "We use teams of five and six, working together," Burt says. "Last year, we created a lot of small projects. At the end of the month, we delivered them to customers and asked, 'What do you think about this?' Sometimes we threw the experimental features away, and sometimes we found they had real value. Just because you have a 10-million-line codebase, that doesn't mean you can't be nimble and quick. You can use Scrum and other Agile techniques with a large codebase — you just need to adjust your thinking a bit. Create a branch, have a small team work on it, see if the idea has merit. There's no substitute for experimenting with the codebase. I think that's changed the way people think about their work. You can have a code jam. And you can, in one day, accomplish something significant in the QuickBooks codebase. And if it only takes 45 minutes to build — why not?"

Most of all, the key to managing a large project was automation. "We automate everything that can be automated," says Burt. "The tools make a huge difference. We maintain all the different versions of QuickBooks, on all our supported platforms, with about 60 code-writing developers. We couldn't do that without automation."

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.