With us today is Steve Hazel, senior director of development at Sauce Labs, a company that focuses on accelerating the Agile web development test process.
Q: Steve, what is functional testing? How does it differ from, say, unit testing?
A: "Functional testing" has always seemed like an unfortunate name to me; "functionality testing" or "feature testing" might be a little clearer. The idea is to verify that the features of an application work end-to-end, from the vantage point of an actual user.
Unit testing, on the other hand, is focused on verifying that a small chunk of code is working as designed, in isolation from other components in the system. So if a unit test fails, you can be reasonably confident you'll know what code you need to look at to fix the problem. But unit tests can't confirm that all these individual components work properly together as a system. That's where functional testing comes in.
A good thumbnail sketch of functional testing is the kind of manual testing almost every software developer has always done: start the program up, and try it out. Push all the buttons, try to use it, see whether anything goes wrong. That kind of test is universal, and it's a simple example of a functional test. Of course, the state of the art these days is automated and much more comprehensive functional testing, but that captures the idea. It's testing at the level of user interactions with software -- when you open Firefox and login to gmail.com to check your email, you're using all the parts of a complex system that makes Gmail work. To verify that the full user experience is working (that the a "units" operate together as a system or "functional" whole), you need functional tests.
Q: Why is functional testing getting so much attention these days?
A: I think the headline is that testing, in general, has been getting more attention amongst software developers, thanks to the adoption of Agile development methods and their focus on increased automated testing early in the development process. Agile originally only focused on unit testing, and unit testing has very quickly become a standard, expected developer practice. Developers have realized, though, that they can't live on unit tests alone. Functional tests are needed as part of what my cofounder Jason Huggins likes to call a "complete breakfast" of software testing.
Q: Can you tell us what Selenium is in general and Sauce RC 1.0 is in particular? What's unique about the Sauce implementation?
A: Selenium is a browser automation tool. It provides commands to control a web browser to do the same kinds of things a user would when using a website. A basic test using Selenium would include commands that tell a web browser to open a webpage, type text into a field, click a button, and verify the value of some result on the page.
Selenium achieves a broad goal. It enables software developers and testers to write functional tests in any programming language (Java, C#, Python, Ruby, PHP, etc) against any kind of website or web application, and run those tests in any web browser (Internet Explorer, Firefox, Safari, Google Chrome, etc).
There are two main components of Selenium: Selenium IDE, a Firefox extension that can record/playback simple tests which affords new users a low-threshold entry to Selenium testing, and Selenium Remote Control (RC), which has more advanced features and supports more languages and more browsers than Selenium IDE.
Sauce RC is our distribution of Selenium RC. It includes the latest release of Selenium RC, plus some patches that we felt were important, and an installer and configuration interface that make it easy to install, configure, and administrate. It has a web interface for configuration, which is a simple way to enable remote administration, and it can connect to our cloud service, Sauce OnDemand, to run tests in browsers that might not be available locally. We'll be open sourcing Sauce RC under the same (Apache) license as Selenium itself, and we contribute our improvements to Selenium RC back to the Selenium project.
Q: How does all this relate to software parallelization and multicore platforms?
A: Well, you really want to be running tests very frequently, and you want them to run in a reasonable amount of time. There's that famous research by Boehm that says the cost of defects increases exponentially over the phases of the software life cycle. So it's worth putting the effort into catching defects earlier. And a comprehensive test suite, especially with functional tests that exercise the entire software stack, can take a while, sometimes many hours, to run.
The good news is, you can throw CPUs at the problem, because well-written tests are naturally parallelizable. Tests that leave behind significant persistent state, or which interact with one another, are more fragile, more prone to false-positive failures when the application is working fine, and less able to isolate failures so that you have some idea where to start looking when a problem is identified. The tendency, if you're doing a good job writing useful and independent tests, will be that they can be parallelized with little or no further effort.
Q: Where does the cloud come into play?
A: It's important in a couple of ways. Parallelization of tests is one obvious benefit. Testing is this bursty process that requires a lot of CPU cycles for a short period of time. Requisitioning CPUs from an elastic cloud makes high-leverage use of the natural capability of the cloud and is an economical way to get testing done rapidly.
But beyond that, I think every software shop that uses Selenium heavily ends up with a bunch of machines that do nothing but run Selenium tests all day. That infrastructure translates into maintenance headaches, and because it's usually not your core focus, it tends not to get built out at the level of quality and with the kinds of features that would make it really great to use. Sauce OnDemand is our answer to those problems. It provides Selenium testing infrastructure as a cloud service, so you can plug it in and get on with the business of building great software. Sauce RC in turn makes that easy. In addition to working as a normal Selenium RC server, it has a mode that connects to Sauce OnDemand. If you have a test you're running in Firefox on your Mac laptop, you can be running it in Internet Explorer 6 in seconds just by selecting that browser from the drop-down menu in Sauce RC.