[This week, I've invited Cédric Beust to pen the editorial. Cédric writes a popular programming blog at beust.com. He's also the author of Next Generation Java Testing, which I've long considered the best available book on developer-based testing. He is a former Google engineer now working on Android. He is also the founder and lead developer of TestNG, the most widely used Java testing framework outside of JUnit. Ed.]
A few years ago, I had a conversation with a coworker that is still etched in my memory. We were talking about a library that he had recently open sourced and that he was very proud of. I had taken a look at it and I had one question for him: "I didn't find any tests in your distribution, are you shipping them separately?" He looked at me as if I had just insulted him. As it turns out, it's exactly how he felt. "Tests? I'm a developer, I don't write tests, I leave that to others."
Admittedly, this happened more than ten years ago. Back in those days, there was still a strong separation between "real" software engineers, who wrote production code, and "lesser" ones, who wrote tests. This certainly sounds laughable today, which is a testament at how far we've come; but once in a while, it's good to remember where we started. Today, you will be hard-pressed to find a single developer who doesn't think that writing tests for their code (and even other people's code) is part of his job. When I interview job candidates, I typically work on two aspects: "Write code to solve this problem" and "Now that you've written this function, how would you test it?" I would never hire someone who is not comfortable with both of these activities.
Testing is now universally accepted as being an integral part of software engineering and we have also created an entire ecosystem around it, with tools, frameworks, methodologies, design patterns, and countless articles, books, and forums dedicated to this very topic. If you want to improve at testing your code, there is no shortage of material.
There is a lot of good advice to be found in this cornucopia, but abundance often comes with confusion, inconsistencies, and even sometimes downright bad recommendations. I'd like to revisit some testing concepts that have emerged over the past decade and try to shed a more critical light on them.
A unit test verifies that a single compilation unit works as expected. It's not supposed to use any other class in your code base and I've seen some people add restrictions, such as not allowing file, network, or database access in the interest of speedy runs and minimalistic dependencies. I think these are good rules of thumb to keep in mind, but you shouldn't be afraid to break them if it makes sense to do so. From a practical standpoint, I have found it made much more sense to categorize a test based on how fast it runs than on what it does.
I also question all the attention that unit tests are receiving at the expense of the other kinds of tests (functional, integration, etc.). The truth is that functional tests serve your users, while unit tests serve you the developer. A unit test is just a convenience that allows you to track down bugs faster. At the end of the day, the reason you write tests is to make sure that your users will be able to be productive with your application, not to make sure that you can debug faster.
When should you write unit tests? And functional tests? These are difficult questions to answer that you should ask yourself each step along the way, because the answer will vary depending on a lot of factors such as where you are in the lifecycle of your product, the team culture, the existing testing infrastructure, and so on.
If there is one thing you should keep in mind about unit tests, it's this: Unit tests are a luxury. Sometimes, you can afford a luxury; and sometimes, you should just wait for a better time and focus on something more essential such as writing a functional test.
Test-driven development (TDD) means well. It really does. It's trying to force people into good design by imposing a specific process on the way they write code. I have found that this approach works well for junior developers by forcing them to think about testing very early on.
I have seen a lot of material on the positive aspects of TDD, so I'll skip those to jump directly to more negative aspects that I have observed.
First of all, I find that TDD encourages code churn. Most of the time, it will take two or three iterations on a new piece of code before reaching a stage where you think you have a good starting point. Unit tests for these early versions are pretty much guaranteed to be refactored constantly with little gain, and just eat a lot of time keeping the compiler happy. If you're going to throw away the first two versions of your code, you might as well skip the first two versions of your tests as well.
TDD code also tends to focus on the very small picture: relentlessly testing individual methods instead of thinking in terms of classes and how they interact with each other. This goal is further crippled by another precept called YAGNI, (You Aren't Going to Need It), which suggests not preparing code for features that you don't immediately need. If your experience tells you that very soon, you are indeed going to need a feature, you might as well add it right away, even if it doesn't quite make sense just yet.
TDD appears very convincing on paper, or more precisely, on slides. It looks like a great way to design a scorecard for bowling a Stack or a List or a Currency class but I have always struggled to find how I can use it effectively in my day job (think mobile applications or GUI's), and I haven't yet found a convincing case where using it clearly outweighs its drawbacks.
Don't fall for the false dichotomy claiming that code that is not created with TDD is necessarily buggy. First, there is a lot of untested code out there that is working just fine; and second, testing "last" instead of "first" is an equally valid approach to this problem.
Engineers like to measure things. Producing and evaluating numbers is a common way to measure progress and plan our work, such as how much of a program has been exercised by tests. This code coverage is derived by running your tests with a tool that keeps track of all the parts of your code that are invoked, and then issuing a report to tell you what areas were not run. It's difficult to argue against the idea, but like all techniques, it's easy to get carried away and lose track of the big picture.
I noticed two specific excesses regarding coverage: focusing on an arbitrary percentage and adding useless tests.
What percentage of your code should be covered before you can ship? 100%? 90%? 80%? You will find a lot of different numbers in the literature and I have yet to find solid evidence showing that any given number is better than another. Obsessing over numbers in the absence of context leads developers to write tests for trivial pieces of their code in order to increase this percentage with very little benefit to the end user (for example, testing getters). You should also keep in mind that these numbers represent static, not dynamic, coverage: 100% code coverage doesn't mean that you are testing everything that can possible happen to your code just that 100% of your code has been run at least once.
Running code coverage tools is a good practice, but you should exert caution when analyzing the reports. I recommend reviewing which pieces of your code base are not being covered and deciding for yourself whether not testing a particular area is reasonable or too risky.
There is a point of diminishing returns in increasing code coverage, so you should always wonder whether adding 1% to that number is more important than, say, improving an existing test case to make it more thorough or adding a new feature.
I have barely touched the surface of some of the testing misconceptions that pervade our profession, and I'd like to end on a positive note: All these techniques have beneficial aspects that you can experience only if you actually try them. Once you've done that, keep in mind that these methodologies are just tools, and that you should consider using them only after carefully examining the problem you are trying to solve and the context that surrounds it.