Channels ▼
RSS

Testing

Unit Testing: Is There Really Any Debate Any Longer?


On a panel several years ago, I was asked what was the greatest benefit that Agile had delivered to me personally. It took me no time to respond "unit testing." (While this answer is not historically accurate — unit testing precedes the Agile movement — it's clear that the Agile exponents made it a widespread practice. In large part, because of Kent Beck's lapidary JUnit implementation, which has been widely copied to most major languages.)

The specific benefit I — and many other developers — have enjoyed is quite simply less time spent in the debugger. Today (and for years now), I write code and then I write unit tests that exercise the edge cases and one or two main cases. Right away, I can tell if I missed something obvious or if my implementation has a slight burble that mishandles cases I expected to flow through easily.

This doesn't mean I can assert that I now feel comfortable my code is right. But what I do know is that it's better because it's been partially tested. The result of this is that the long sessions in the debugger trying to find why, for example, some data packets have an unexpected zero in the last byte, are pretty much done with. An error like that — invalid values or an improper format — will be caught right away by my tests. Previously, I'd have to wait for the bug to surface, then trap it, back up through code, examine and test it, and keep following the data up the call chain only to discover that at the very start when I created a substring, I miscounted by one the number of bytes to extract. Those days of 3 AM debugging are gone. Most of my debugger sessions today are about logic errors that transcend the unit level. I am happier and more productive.

I also enjoy another considerable advantage that I did not have prior to unit tests. As the collection of tests grows, the tests themselves become monitors of sorts that watch over my code to make sure it has not shifted unexpectedly beneath me. Because my code coverage is in the 75%-80% range, I have a lot of these monitors watching out for me. I run them all as a regression sequence any time I make a change that might have more than local impact. When I make a big change, I can tell right away if I have unhooked anything. For example, the other day, I changed the name of a major package. There were references to it throughout my code base. I had the IDE do the refactoring. When the compilation worked the first time, I felt a first measure of comfort. When all the unit tests passed, I had a high level of confidence that the change had been implemented non-disruptively. Without unit tests, I would never have made the change, because of the risk that it would cause problems for me far down the chain.

These benefits are not unique to me. Many Agilists view tests as enabling flexible development. In fact, the edgier subset among the Agilists won't code at all unless they have tests written beforehand. This approach is called Test-Driven Development (TDD), which they swear by, but which does not appeal to me. Regardless, the fundamental recognition that by writing unit tests, you assure yourself of the ability to make major changes easily is indeed a universal recognition.

However, the recognition is not universally embraced. A reader recently wrote to me, grousing about the current vogue that so values unit tests. His view was that the test are a waste of time. I've heard this view before. Unit testing has not been a waste of time for me, because the time it costs to write the tests is recovered from the shortened debugging cycle. And I gain the ability to make changes to the code with confidence. But the classic complaints often point to exactly such large changes. Namely, that if — in the Agile spirit — you throw away a bunch of code because of a shifting requirement, you also have to throw out a bunch of unit tests. Because unit tests typically represent 50%-75% of the size of the code base, you're chucking 1.6x code, say, for every 1x that needs to be changed. This is true, of course, but there's no way out of that bind. I could write untested code that I dreaded to debug and then I could feel better about having to dump it. But that's not exactly a happy place either.

One place where the complaint rings true is psychologically. The more code you've written, the more attached you become to it and the less you're willing to throw it away. And tests only add to this resistance. But the reality is that code is rarely stripped off and dumped en masse. Rather, some pieces are replaced selectively and others are modified. In this process, tests once again help assure that the code works as expected.

Most organizations today use unit tests in varying degrees. In a recent talk, Jeff Atwood of Coding Horror fame said he uses tests selectively — mostly when he's testing package interfaces. Others, as I pointed out, are more committed to their use throughout a project. If you're among the hold-outs who don't use unit tests at all, you're missing an excellent opportunity to make your code better and to improve your coding experience. As I said earlier, of all the practices the Agile revolution has brought to the mainstream, this is probably the one that will save you the most time and aggravation.

— Andrew Binstock
Editor in Chief
alb@drdobbs.com
Twitter: platypusguy


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Comments:

ubm_techweb_disqus_sso_-5c471a50164a7b307706e8eba25b8488
2012-10-23T16:54:13

Good question. I have been trying to figure this out. For the tools I use personally (mainly .net development), we are not there. I don't have access to good automated proof tools. However, it is done in some industries, such as avionics, where better tools are available.


Permalink
ubm_techweb_disqus_sso_-a80544abb0e401ec545e4f2ece9399c3
2012-09-25T20:38:19

I don't think that the argument is valid; due to changing requirements you have to throw away a large portion of the code base (i.e. unit tests).

The reason why you are throwing it away is because it is wrong entirely to satisfy the requirement of the business domain. It was our fault that we did not get it correct in the first iteration.

I believe that the person that said that to you is essentially saying, "Due to the fact of engraving quality in your code, if it changes we have to redo the same quality and I don't want to."

It just seems to me like a void argument.

All the benefits are not realized until you are in an incredibly complex situation of people, technical requirements and business requirements. It puts the "care" into your product. Also most people don't realize the potential of TDD because generally speaking until your application grows to an incredibly complex state that not one single person can just "remember" how things should work. It's the reason why computers exist so we don't have to remember to go through lists items to make sure they work. What happens when this list takes a programmer 2 days to go through?

Unit tests prevent this. There's actually a case on DotNetRocks.com of someone speaking of a problematic case: what happens when your tests take days to run?

In my blog I wrote about TDD:

"In the case of not executing TDD at the start of a project the process will be as follows; the customer ends up finding something that they think is a problem, calls support, then someone needs to determine whether it is a problem, send to development, development schedules it and solves the immediate symptom and usually not the root cause in which case opens up 5 other untested paths of code and so on! Usually Cowboy Coding is happening the the environment as well. Have fun! If you do not do TDD explain this to your customer immediately. Actually don't you will likely lose them! Instead start doing TDD!"


Permalink
ubm_techweb_disqus_sso_-5ddb47b0783a9f9bb2b3241743d1d1df
2012-09-12T23:00:24


Unit testing object-oriented code is not cost-effective because, when done right, it is a great deal of work. This is partly the result of our toilsome modern software development process, but it also stems from the fact that much OO code is not amenable to unit testing. Many people claim to do unit testing and find it helpful and not overburdening. Most of those people are not unit-testing as it needs to be done. What they do is a repeatable form of developer testing, aided by software products marketed as unit testing tools.

“Much OO code is not amenable to unit testing” means that not much of it fits the ideal scenario in which the behavior of an element (method, function, subroutine) of a module is wholly determined by its arguments, and its effects wholly contained within its return value. On the contrary, these elements work together, sharing information encapsulated by the unit as a whole. This presents challenges to unit testing because bugs don’t always bite the elements they infest. Sometimes they produce unintended effects in other parts of the unit. Sometimes effects are delayed even. Historically, a single unit test case exercises one element of a module—this keeps it simple. The idea is that the unit can be completely tested by a collection of such narrowly-focused test cases. But the move to object-oriented design—to encapsulation, in particular—made this an idea whose time has passed.

“Unit-testing as it needs to be done” means: done in a way that enables you to “take credit” for it. That is, done in a way that would satisfy an auditor from one of the regulatory agencies. If I’m an auditor, I will insist on a number of things that you might find daunting:

1) A unit test shall not be written by the same person who coded the unit-under-test (UUT). If the coder employed any unstated assumptions in coding, he will almost inevitably repeat those assumptions when writing the test. You can see how this could lead to problems.

2) A unit test shall be written without looking at the code to be tested. You’re testing whether the unit meets it specifications, that is, what the design says it should do. If you look at the code, your tests will address the coder’s choices, not the designer’s choices. You will test what the code does, not of what it should do. This implies that the behavior of anything you want to test must be specified in writing somewhere.

3) Unit testing may involve hacking things in the code surrounding the UUT, in the test harness, but there shall be no hacks within the UUT– nothing compiled in or out of the UUT based on something like #ifdef UNIT_TESTING.

4) Unit tests themselves shall be reviewed by other developers, and the unit tests themselves shall be verified. It must be proven, and documented, that a unit test will not say a unit passes when it should not. If a unit test is modified, it must be reviewed and verified again.

5) Any tool used to produce unit tests, execute them, or report their results, shall be verified for its intended use. As the auditor, maybe I’ll accept the tool manufacturer’s certification of verification, but maybe I won’t if I think you are using the tool in an unusual way.

Earlier I said that unit tests should be written without looking at the code. This is also the MO of TDD (test-driven development): write the tests against the design, before any code is written. In theory, that’s not a bad idea. But as is usually (always?) the case, theory and practice differ. How so?

Much of the effort you put into writing pre-coding units tests will be wasted, because, just as no battle plan survives contact with the enemy, no design survives contract with coding. Heck, not even your requirements will survive contact with coding. Attempting to perfect your requirements and your design before commencing coding will delay the start of coding a very long time, and you will still fail to see your design preserved. This is because coding is the only rendering of your ideas precise enough to show you the dead-ends.

A detailed design that is not a house of cards is possible, however, after the coding of a prototype has allowed you to see all the way to the bottom. Regard the prototype as a learning experience. Lay it aside, but take what you learn from it to revise your requirements and to inform a detailed design that you can now write with complete knowledge of the destination. In my experience, this is the fastest way to get good software.


Permalink
ubm_techweb_disqus_sso_-86a35708096e1ce604b89e4003d1df8f
2012-09-12T21:25:31

Early 1980s if not earlier? Definitely earlier. I wrote numerical analysis software back in the late 70s, and unit tests (we called them regression tests) were integrated in our build environment in pretty much the same way we do it now.

And it was definitely not original with me - I copied the basic idea from the IMSL routine library, which was a FORTRAN library with unit tests and results for a huge range of platforms embedded in the code as comments. There was some utility you ran to extract the tests and run them on your platform.


Permalink
ubm_techweb_disqus_sso_-9648df76fc3e3f0cc991cd354e0812fd
2012-09-12T19:18:34

I understand TDD, though I subscribe to incremental testing as I develop. The tests become outdated and wrong in this instance, but I compile and test frequently in this manner. A normal daily code session could last for 6-8 hours easily and all that time I am either writing code building the product or I am writing tests to test the individual pieces and eventually integrating those pieces together.

Something that I dont get though is a fundamental lack of understanding of code that one has written. If you write in your "style" all the time, then it doesnt matter when you come back to your code. Maybe one is speaking about writing someone whom has yet to develop a style of programming. I prefer to write code that looks like it was automatically generated from a configuration. That results in a clean implementation.

Additionally, I am refactoring my code continuously as I write it. I will end up writing a multiple pieces of code multiple times until I believe that it is ready enough. For me ready enough is for maintenance. The code should be written in such a way as to minimize long term maintenance. This has frequently allowed for minimal maintenance over time. Understanding the complexity of code and reducing the complexity as much as possible eliminates bugs from code immediately. Having nested loops and if statements and other control structures increases the complexity and thus the bugs. Unless the complexity gives a large performance boost, complexity is the bane of software in my opinion.

However, the golden egg of software development is incremental testing. Unit testing is nice but I consider incremental testing distinctly separate from unit testing. I get productivity and speed of testing and can potentially even reuse the incremental tests as the product driver.

I have done the unit test thing and it is nice to have, but it is just as error prone as the code. With incremental testing, you're testing immediately while developing and it points out fallacies in one's design and development and allows you to take step back and refactor to fix the problem(s). The more generic and reusable the solution, the better the solution.

Lastly, build each piece as separate components and support systems. Build your interfaces, build your parsing, build the grassroots before you build the organization. If you build your grassroots before the organization, then you can repurpose them for multiple organizations. If you build the organization first, then the grassroots you build are specific to the organization. So build the grassroots and then build the organization. This makes the best use of code. Make each grassroots components do its part the best it can. The organization can then deploy the grassroots into multiple different situations as the grassroots become reusable.

Just my two cents.


Permalink
ubm_techweb_disqus_sso_-cc156d1367e705123ded4a5a2b014189
2012-09-12T15:21:12

doing TDD for only the resulting tests is like going jogging only for the view.


Permalink
ubm_techweb_disqus_sso_-7ac51850d33a29b19628f97e97f4b706
2012-09-12T12:15:20

Another huge advantage of TDD is the fact that you basically can't easily test bad code (or even hacks).. Therefore any code that is testable implies a minimum amount of sourcecode quality.

Also, I nearly always find some quirks in the original code when I write tests (I am honest). If this happens, the tests are already useful.

And its a neat way to reproduce bugs and write regression tests for them. With C++, the overhead of integration testing (aka the full app in test mode) is very, very low.


Permalink
ubm_techweb_disqus_sso_-6d6ce254ed0d97760f265139f5e4fb9c
2012-09-12T03:41:25

nm


Permalink
ubm_techweb_disqus_sso_-c5002f6ed915245ad66a03bed1c4f621
2012-09-12T03:24:59

If the tools are capable and their cost comparable to that of the developer(s), then that seems to be a given.

Do you think we're there yet? I'm assuming that you're not talking about "just" automatic test generation.


Permalink
ubm_techweb_disqus_sso_-c5002f6ed915245ad66a03bed1c4f621
2012-09-12T03:15:11

Not. I agree with the author that TDD is still very viable debate bait.

TDD is great for organizations that are proven incapable of any sort of project management or self-discipline. If you can't have tests begun or developed concurrently by someone very close by, expect to spend a big chunk of your development schedule on "integration". TDD is the last resort for organizations before they simply have their keyboards taken away altogether. Therefore If you can't test your code while it's still fresh in your mind (a worthy goal), you need the diaper that is TDD. And enjoy all the refactoring and redesign rewarded by that your lack of planning.

As @DNMurphy said below, unit testing has been around since *at least* the early 80s. I'm unsure if we called it "unit testing" back then or "not pushed into Marketing or Management". Actually, it just fell into the greater category of "test coverage". In thirty years we now have GUIs on tools of a barely improved class than we used (or developed once and re-used to death internally) back then. So maybe it has, in fact, taken thirty years for the idea of unit/acceptance testing to catch on.

It's sick that this is still worthy of discussion in this century. Wow. Just... wow.

| sed 's/unit testing/continuous integration/g' while we're at it.


Permalink
ubm_techweb_disqus_sso_-5c471a50164a7b307706e8eba25b8488
2012-09-11T23:19:59

Given a fixed budget and a choice between a proof of correct behavior, and more run-time tests, the proof is a better deal, if you can automate it so that it can be re-run.


Permalink
ubm_techweb_disqus_sso_-3e2fc29a0efe0d562046649a489dd8cc
2012-09-11T22:24:09

I would agree that 100% coverage is not economically sound on a legacy project but adding them as you bugfix yields virtually the same benefit -- you know when "fixed" bugs are reintroduced. When you undoubtedly break something else, you add more tests. On a fragile system, you get your coverage as you're adding features & fixing bugs.

I disagree that it should not be a technical decision. It costs you nothing to document your work via unit tests (assuming you're actually running the code you develop).


Permalink
ubm_techweb_disqus_sso_-96fdfc3b41d4f4a602ffaf9641880de3
2012-09-11T22:17:30

No responsible agile advocate should insist on retrofitting an entire legacy system with unit tests. You write them for the area you are going to change. That should not be very expensive, and the alternative - to make significant changes without fast test backing - is almost certainly going to be more expensive.


Permalink
ubm_techweb_disqus_sso_-d67fa7836f3627e9256d451aeeb659d0
2012-09-11T21:47:23

An exection would be large (multi million line) legacy systems with code dating back to the 1980's or earlier. Retro fitting unit tests to such a system, typically not written with testability (as we know it today) in mind is not commercially practical. Many a UT Evangelist has insisted that it can be done cheaply and try. The problem is that in these systems it is hard to target the edge cases and too easy to focus on the obvious. As a result of needing results quickly, to prove the benefit of UT, the obvious is covered, and the edge case defects keep rolling in att he same rate. Mangement look at the stats and rightly declare it a waste of time and money - i.e. failure.
To retro-fit unit tests to code designed before TDD was a TLA is expensive and time consuming. It's probably a cheaper option than a complete rewrite, but in that same order of magnitude. The mamnement team need to consider the future life span of the product and decide if the effort required to implement UT is worth the additional cost.

i.e.Implementing TDD/ Unit Testing should be a commercial, not a technical decision. Evidence points to the cost benefit in strong favour of unit tests for new projects. Evidence of the benefit of retro fitting unit tests to legacy projects is scarce.


Permalink
ubm_techweb_disqus_sso_-96fdfc3b41d4f4a602ffaf9641880de3
2012-09-11T21:25:06

Unit testing is now so popular that... our testing group calls all automated tests, "unit tests."

Yes, there are plenty of obstacles to adoption of what agilists call "unit tests," - among them the skepticism from so many developers that it is possible to write tests that take only 10 msec or more, that you don't need to go end-to-end most of the time, and so on.

There is actually often a world of difference between the kinds of tests you create using TDD and the tests you write after the fact. Still, given the incredibly low adoption rate of TDD I would be thrilled to find legacy code that has after-the-fact fast unit tests. Usually, what I find are tests that average multiple seconds if not minutes long.


Permalink
ubm_techweb_disqus_sso_-f5556c08afb65764ed6cd9f93eab3cf0
2012-09-11T21:04:36

I agree-- with one major exception: user interfaces. The good developers who still don't like unit tests tend to write strongly UI-driven apps, where nearly the entire code base can be tested with a carefully planned feature walk-through.

The fundamental question is whether or not code correctness is determined by a well-defined property that is known in advance. That's not purely the purview of user interfaces, though other examples are more rare. One way to tell if your code isn't a perfect candidate for unit testing is if a bug ever turns out to be better than the intended design. (This happens all the time if you're writing a screen saver.) A buggy high frequency trading application which makes more money than the intended design (e.g. by skipping an unnecessary, time-consuming loop) would be one example. I've also done a number of AI-ish apps where it's nearly impossible to tell if the (non-deterministic) code is running properly, since the overall program runs fine. (And yes, there are a few workarounds: use a mock random number generator, set the seed explicitly, or have the unit test loop a zillion times. All of these are fragile.)

There are also some UIs that I wouldn't think of deploying without unit tests: in particular, where the user is relying on the computer to be correct. Think of a bank website, where 99% of the code is UI, but the 1% where it shows your account status had better be 100% correct: if you make a deposit, and the balance is cached from two seconds ago, the user may have a heart attack.


Permalink
ubm_techweb_disqus_sso_-9323c3011892751b33193b386c38021f
2012-09-11T21:00:46

Unit testing was normal practice when I was a programmer i the 1980s and there were test harnesses and other tools to aid testing. I worked on numerous sites and nowhere was it acceptable for a programmer to not do unit testing. It was documented and frequently reviewed.

I fail to understand anyone who says unit tests are a waste of time, I caught numerous bugs in my own code with thorough testing, and I am sure I was not alone. Walkthroughs, review and inspections were all common then as well.


Permalink
rudmerriam
2012-09-11T20:40:27

Rename it from Unit Testing to Unit Development. The renaming is very valid. I use TUT with C++ development and JUnit for the small amount of Java that I do. I really do consider the process as development since the units and the framework provide the infrastructure for developing the code. The test almost becomes secondary.


Permalink
ubm_techweb_disqus_sso_-cc156d1367e705123ded4a5a2b014189
2012-09-11T20:34:31

please rewrite this article as:
"TDD: Is There Really Any Debate Any Longer?"


Permalink

Video