Unit Testing in C: Tools and Conventions

In this article, I look at unit tests using two unit test harnesses that work in C. Along the way, I will also discuss some of the common terminology of automated unit testing. Let me start by discussing the fundamental tool, the test harness.

What Is A Unit Test Harness?

A unit test harness is a software package that allows a programmer to express how production code should behave. A unit test harness's job is to provide these capabilities:

• A common language to express test cases
• A common language to express expected results
• Access to the features of the production code programming language
• A place to collect all the unit test cases for the project, system, or subsystem
• A mechanism to run the test cases, either in full or in partial batches
• A concise report of the test suite success or failure
• A detailed report of any test failures

I'll shortly look at two popular harnesses for testing embedded C.They are both easy to use and are descendants of the xUnit family of unit test harnesses.

First, I'll employ Unity, a C-only test harness. Later, I will use CppUTest, a unit test harness written in C++, but not requiring C++ knowledge to use. You'll find that the bulk of the material in this article can be applied using any test harness.

Here are a few terms that will come in handy while reading this explanation:

• Code under test is just like it sounds; it is the code being tested.
• Production code is code that is (or will be) part of the released product.
• Test code is code that is used for testing the production code and is not part of the released product.
• A test case is test code that describes the behavior of code under test. It establishes the preconditions and checks that significant post conditions are met.
• A test fixture is code that provides the proper environment for a series of test cases that exercise the code under test. A test fixture will assist in establishing a common setup and environment for exercising the production code.

To take the mystery out of these terms, let's look at a few tests for something we've all used: sprintf. For this first example, sprintf is the code under test; it is production code.

sprintf is good for a first example because it is a standalone function, which is the most straightforward kind of function to test. The output of a standalone function is fully determined by the parameters passed immediately to the function. There are no visible external interactions and no stored state to get in the way. Each call to the function is independent of all previous calls.

Unity: A C-Only Test Harness

Unity is a straightforward, small unit test harness. It comprises just a few files. Let's get familiar with Unity and unit tests by looking at a couple example unit test cases. If you are a long-time Unity user, you'll notice some additional macros that are helpful when you are not using Unity's scripts to generate a test runner.

A test should be short and focused. Think of it as an experiment that silently does its work when it passes, but makes some noise when it fails. This test checks that sprintf handles a format spec with no format operations.

TEST(sprintf, NoFormatOperations) 
{ 
    char output[5]; 
    TEST_ASSERT_EQUAL(3, sprintf(output, "hey")); 
    TEST_ASSERT_EQUAL_STRING("hey", output); 
}

The TEST macro defines a function that is called when all tests are run. The first parameter is the name of a group of tests. The second parameter is the name of the test. We'll look at TEST in more detail later.

The TEST_ASSERT_EQUAL macro compares two integers. sprintf should report that it formatted a string of length three, and if it does, the TEST_ASSERT_EQUAL check succeeds. As is the case with most unit test harnesses, the first parameter is the expected value.

TEST_ASSERT_EQUAL_STRING compares two null-terminated strings. This statement declares that output should contain the string "hey": Following convention, the first parameter is the expected value.

If either of the checked conditions is not met, the test will fail. The checks are performed in order, and the TEST will terminate on the first failure.

Notice that TEST_ASSERT_EQUAL_STRING could pass by accident; if the output just happened to hold the "hey" string, the test would pass without sprintf doing a thing. Yes, this is unlikely, but we better improve the test and initialize the output to the empty string.

TEST(sprintf, NoFormatOperations) 
{ 
    char output[5] = ""; 
    TEST_ASSERT_EQUAL(3, sprintf(output, "hey")); 
    TEST_ASSERT_EQUAL_STRING("hey", output); 
} 

The next TEST challenges sprintf to format a string with %s.

TEST(sprintf, InsertString) 
{ 
    char output[20] = ""; 
    TEST_ASSERT_EQUAL(12, sprintf(output, "Hello %s\n", "World")); 
    TEST_ASSERT_EQUAL_STRING("Hello World\n", output); 
} 

A weakness in both the preceding tests is that they do not guard against sprintf writing past the string terminator. The following tests watch for output buffer overruns by filling the output with a known value and checking that the character after the terminating null is not changed.

TEST(sprintf, NoFormatOperations) 
{ 
    char output[5]; 
    memset(output, 0xaa, sizeof output); 
    TEST_ASSERT_EQUAL(3, sprintf(output, "hey")); 
    TEST_ASSERT_EQUAL_STRING("hey", output); 
    TEST_ASSERT_BYTES_EQUAL(0xaa, output[4]); 
} 
TEST(sprintf, InsertString) 
{ 
    char output[20]; 
    memset(output, 0xaa, sizeof output); 
    TEST_ASSERT_EQUAL(12, sprintf(output, "Hello %s\n", "World")); 
    TEST_ASSERT_EQUAL_STRING("Hello World\n", output); 
    TEST_ASSERT_BYTES_EQUAL(0xaa, output[13]); 
}

If you're worried about sprintf corrupting memory in front of output, we could always make output a character bigger and pass &output[1] to sprintf. Checking that output[0] is still 0xaa would be a good sign that sprintf is behaving itself.

In C, it is hard to make tests totally fool-proof. Errant or malicious code can go way beyond the end or way in front of the beginning of output. It's a judgment call on how far to take the tests. You will see when we get into TDD how to decide which tests to write.

With those tests, you can see some subtle duplication creeping into the tests. There are duplicate output declarations, duplicate initializations, and duplicate overrun checks. With just two tests, this is no big deal, but if you happen to be sprintf's maintainer, there will be many more tests. With every test added, the duplication will crowd out and obscure the code that is essential to understand the test case. Let's see how a test fixture can help TEST cases.

Test Fixtures in Unity

Duplication reduction is the motivation for a test fixture. A test fixture helps organize the common facilities needed by all the tests in one place. Notice how TEST_SETUP and TEST_TEAR_DOWN keep duplication out of the sprintf tests.

TEST_GROUP(sprintf); 

static char output[100]; 
static const char * expected; 
TEST_SETUP(sprintf) 
{ 
    memset(output, 0xaa, sizeof output); 
    expected = ""; 
} 

TEST_TEAR_DOWN(sprintf) 
{ 
} 
static void expect(const char * s) 
{ 
    expected = s; 
} 
static void given(int charsWritten) 
{ 
    TEST_ASSERT_EQUAL(strlen(expected), charsWritten); 
    TEST_ASSERT_EQUAL_STRING(expected, output); 
    TEST_ASSERT_BYTES_EQUAL(0xaa, output[strlen(expected) + 1]); 
}

The shared data items defined after the TEST_GROUP are initialized by TEST_SETUP before the opening curly brace of each TEST. The data items comprise file scope, accessible by each TEST and all the helper functions. For this TEST_GROUP, there is no cleanup work for TEST_TEAR_DOWN.

The file scope helper functions, expect and given, help keep the sprintf tests clean and low on duplication.

In the end, it's just plain C, so you can do what you want as far as shared data and helper functions. I'm showing the typical way to structure a group of tests with common data and condition checks.

Now these tests are focused, lean, mean, and to the point.

TEST(sprintf, NoFormatOperations) 
{ 
    expect("hey"); 
    given(sprintf(output, "hey")); 
} 

TEST(sprintf, InsertString) 
{ 
    expect("Hello World\n"); 
    given(sprintf(output, "Hello %s\n", "World")); 
} 

Notice that once you understand a specific TEST_GROUP and have seen a couple examples, writing the next test case is much less work. When there is a common pattern within a TEST_GROUP, each test case is easier to read, understand, and evolve, as change becomes necessary.

Installing Unity Tests

It is not evident from the example how the test cases get run with the necessary pre- and post-processing. It's done with another macro: the TEST_GROUP_RUNNER. The TEST_GROUP_RUNNER can go in the file with the tests or a separate file. To avoid scrolling through the file, I use a separate file. For the two sprintf tests written, the TEST_GROUP_RUNNER looks like this:

#include "unity_fixture.h" 
TEST_GROUP_RUNNER(sprintf) 
{ 
    RUN_TEST_CASE(sprintf, NoFormatOperations); 
    RUN_TEST_CASE(sprintf, InsertString); 
} 

Each test case is called through the RUN_TEST_CASE macro. Essentially, this RUN_GROUP_RUNNER calls the function bodies associated with each of these macros:

TEST_SETUP(sprintf); 
TEST(sprintf, NoFormatOperations); 
TEST_TEAR_DOWN(sprintf); 

TEST_SETUP(sprintf); 
TEST(sprintf, InsertString); 
TEST_TEAR_DOWN(sprintf); 

Invoking TEST_SETUP before each TEST means that each test starts out fresh, with no accumulated state. TEST_TEAR_DOWN is called to clean up after each test.

Now that the tests are wired into a TEST_GROUP_RUNNER, let's see how the TEST_GROUP_RUNNERs are called. For this last step, we have to look at main. You will have a main for your production code and one, or more, for your test code. The Unity test main looks like this:

#include "unity_fixture.h" 

static void RunAllTests(void) 
{ 
    RUN_TEST_GROUP(sprintf); 
}
 
int main(int argc, char * argv[]) 
{ 
    return UnityMain(argc, argv, RunAllTests); 
}

RUN_TEST_GROUP(GroupName) calls the function defined by TEST_GROUP_RUNNER. Each TEST_GROUP_RUNNER you want to run as part of your test main has to be mentioned in a RUN_TEST_GROUP. Notice that RunAllTests is passed to UnityMain.

One unfortunate side effect of using a C-only test harness is that you have to remember to install each TEST into a TEST_GROUP_RUNNER, and the runner is invoked by calling UnityMain. If you forget, tests will compile, but not run — potentially giving a false positive.

Because of this opportunity for error, the designers of Unity created a system of code generators that read your test files and produce the needed test runner code. To keep the dependencies low for getting started with Unity, I've opted to not use the code-generating scripts and manually wire all the test code.

When I discuss CppUTest later in this article, you will see another solution to that problem. But before doing that, let's look at Unity's output.