Automating QA

The end result of the build is a set of CD-ROM packages and downloadable packages for the Web. At that point, a custom-built app called AutoLab takes control. AutoLab sets up virtual machines and installs the various releases. First it preps the machines across multiple versions of Windows. Then it runs the installation, opens a company, and closes a company. That confirms that the build resulted in code that installs and runs. That's the basic acceptance test.

Once the code is loaded, the team runs feature scenarios and regression tests. For example, they may exercise all the options of the invoice system. This is all done with Micro Focus SilkTest. Intuit relies upon an outside team to maintain these tests. "You don't put your top talent on maintaining test scripts," Burt notes, "but you must have someone who is responsible for maintaining them. That's a natural thing to outsource."

The results of functional testing are delivered to developers every morning. Assuming the code works properly, developers sync their systems to the new codebase.

Unit testing isn't a big part of QuickBooks for Windows — the bulk of the codebase was written before unit testing was acknowledged as a best practice. QuickBooks Online, which is written in Java, incorporates more unit tests. In fact, Burt says, unit tests dominate the build cycle.

Burt is devoting significant resources to code quality this year. "We've sent multiple teams of five developers into the Coverity logs," he says. "They track down the warnings and fix hundreds and hundreds of minor coding issues. We're cleaning up little things that have been there forever. The goal is to ship a release with zero defects." This isn't just a matter of tools, Burt acknowledges. Team and management support are essential because they must remain committed to QA every day. "It doesn't matter what tools you use," Burt says. "If your team doesn't support it, if they're focused on adding features, you're wasting your time. If you don't want to use a tool like Coverity to track down defects, that's fine — don't use it. But if you say you want it, then you've got to commit. The payoff in code quality is tremendous."

Developers can be resistant to detailed analysis of shipping code at first, Burt says. But Coverity can surprise even senior architects sometimes. "They'll say, 'That's not a bug,'" Burt says. "And then they're like, 'Oh…I didn't think about that path.'"

Burt has worked with the development group to ensure that neither stigma nor penalties apply to developers who find code bugs during detailed analysis. "No one tracks the number of bugs attributed to you," he says. "That's something Intuit has never done. You break the build enough times — well, that gets people's attention. But not bugs. We're focused on eliminating defects, not apportioning blame."

Bug-tracking is handled with Serena Software's TeamTrack. "It's really overkill," Burt observes. "Most of Intuit's other teams have moved to lighter-weight tools, and we may do the same at some point."

Lessons Learned

Dealing with such a large codebase has been an education, Burt acknowledges. "The size of the codebase has spawned a' 10 Commandments of Things Thou Shalt Do Before Checking in Code,'" he says. "It's common sense stuff, but it's important. You're going to do a local build. You're going to run positive and negative tests. You're going to find someone to buddy-check it. If it's a bug fix, the comments are going to refer to the bug. Stuff like that."

Branching, Burt says, is a particular danger. "We have gone branch-crazy in the past," he says. "Everybody wanted his own branch for specific projects. That's great until you want to integrate the various branches back into the main system. If you wait too long and try to roll up all at once, you are in for an ugly month." In the past, Burt says, when a build took 4 hours instead of 45 minutes, it took a half-day to see if a branch broke the code. ElectricAccelerator was a game-changer in that regard. But still, Burt says, more than 5 or 6 branches is probably too many.

One surprising lesson is that small teams work, even for very large codebases — especially, Burt says, in sustaining an entrepreneurial, creative culture. "We use teams of five and six, working together," Burt says. "Last year, we created a lot of small projects. At the end of the month, we delivered them to customers and asked, 'What do you think about this?' Sometimes we threw the experimental features away, and sometimes we found they had real value. Just because you have a 10-million-line codebase, that doesn't mean you can't be nimble and quick. You can use Scrum and other Agile techniques with a large codebase — you just need to adjust your thinking a bit. Create a branch, have a small team work on it, see if the idea has merit. There's no substitute for experimenting with the codebase. I think that's changed the way people think about their work. You can have a code jam. And you can, in one day, accomplish something significant in the QuickBooks codebase. And if it only takes 45 minutes to build — why not?"

Most of all, the key to managing a large project was automation. "We automate everything that can be automated," says Burt. "The tools make a huge difference. We maintain all the different versions of QuickBooks, on all our supported platforms, with about 60 code-writing developers. We couldn't do that without automation."