Channels ▼
RSS

Database

Testing Complex Systems


Everybody knows that unit tests are important and you should strive to test every piece of code you write. Test-driven development (TDD) proponents even believe that you should write the tests first and use them as a design tool. Unfortunately, for most teams and systems, the real world rudely intervenes and automated test coverage is far from complete (including unit tests, integration tests, and full system tests).

The reasons for this apparent neglect of a best practice are diverse. The shocking and painful truth for test aficionados is that successful complex systems can be developed without adequate automated testing. The costs may be higher and necessitate more manual testing, but it is possible. Even the most test-oriented development teams will agree that their code is under-tested (except maybe in some life-critical and mission-critical industries). In this article, I drill down and explain how to write tests for traditionally difficult-to-test areas of the code. As you'll see, testing complex systems has a lot to do with the architecture and design of your code. This is also true for other aspects of software systems such as scalability, performance, and security. You can't just bolt them on top of a horrible mess of code. You have to design testability from the get go or refactor toward it. The good news is that good design (modular and loosely coupled elements with well-defined responsibilities and interfaces between modules) leads to systems that are more testable, scalable, performant, and secure.

The Basics

Developing complex software often involves multiple teams — possibly not even colocated and not in the same time zone. You must have a solid development process that involves source control, an automated build/deployment system, and automated tests. A major concern in such an environment is how to avoid breaking the build, because a broken build caused by one developer can block every other developer. A broken build includes code that doesn't behave as expected. For example, if I introduce a bug to the data access layer, every piece of code that tries to access the data will fail. The more complex the system, the harder it is to gauge the impact of a particular change (this can be minimized by good design). Automated tests become critical here. If all the tests pass after you make your change, you can be pretty sure that you didn't break anyone else's code (assuming reasonable test coverage).

MindReader

For demonstration purposes, I use a C# application, called "MindReader," that reads your mind and tells you what you are thinking about. Figure 1 shows what it looks like in action.


Figure 1.

It has a single button for initiating mind reading, a progress bar to report the progress, and an output area to tell you what you are thinking about. There can be only one mind reading going on any one time, so the button is disabled while mind reading is occurring. It's a pretty simple app, but it allows us to explore sophisticated testing techniques.

Third-Party Code

In a complex system, your team may write only part of the code. You may use open-source libraries, licensed code from external vendors, and (most commonly) code from other teams. As a rule of thumb, you shouldn't create tests for code you didn't write. You normally use third-party code through an API. In some cases, you just use the API directly in your code. When operating in this mode, the third-party is equivalent to built-in libraries for your programming language or the OS. Often however, raw API access is not the best approach. The API may be very complicated, it may not be stable across versions, it may be too low-level, it may be a C API while your system is implemented in C++. In all these cases, teams can elect to write a wrapper around the third-party code that exposes the required functionality, then use the wrapper. These wrappers can be convenient for testing, too, because (if designed properly) they're easy to mock. Sometimes, mocking third-party code is the main justification for writing a wrapper.

When writing wrappers, I recommend that you create a very thin wrapper only around the functionality you need. The goal is to make sure you don't introduce any bugs or performance issues with your wrappers. If the API is large and changes often, look into generating the wrappers automatically if possible. You shouldn't need to write dedicated tests for these wrappers. You may later write another layer on top of the thin wrapper to adapt the API to your application needs, but by then, you are already back in the domain of your code and the access layer can be tested using a mocked version of the thin wrapper if needed.

The only other concern about third-party code is that if you mock it, you need to really understand its behavior — especially for error cases. If an API method returns an error status or throws an exception and your mock doesn't mimic it, you didn't test your code properly against the API.

Another realm of third-party code involves frameworks and plugin-containers. In these cases, you should develop your code to be as host/container-agnostic as possible and test it separately.

The MindReader UI uses Windows Presentation Foundation (WPF) as third-party code and also as the framework where the application starts executing. WPF is usually coupled to your code via event handlers, which may be considered loose coupling, but it means you must deal with WPF signatures and data types (so when a button is clicked, your event handler gets a System.Windows.RoutedEventArgs object that it needs to handle).

The IMainWindow interfaces abstracts all the operations the MindReader app performs on the main window:

public interface IMainWindow
{
	void EnableGoButton(bool enable);
	void SetThoughtText(string thought);
	void UpdteProgressBar(int percent);
}

The IMainWindowEvents interface abstracts all the relevant events that the main window generates and the application wants to respond to:

{
	void OnGoButtonClick();
	void OnKey(string key);
	void OnClose();
}

Note that these interfaces are completely WPF-agnostic. You can switch the UI technology at any time or mock it for testing purposes, as you'll see later. The MainWindowWrapper class (see Listing One) implements the IMainWindow interface and forwards calls to the actual MainWindow WPF class. It also listens to certain WPF events (via event handlers) and forwards them to its IMainWindowEvents sink.

Listing One

using System;
using System.Windows.Controls;

namespace MindReader
{
    class MainWindowWrapper : IMainWindow
    {
        MainWindow _window;
        IMainWindowEvents _sink;

        Button _goButton;
        public MainWindowWrapper(MainWindow window)
        {
            _window = window;
            _goButton = (Button)_window.FindName("goButton");

            // Hook up event handlers
            _goButton.Click += new System.Windows.RoutedEventHandler(_goButton_Click);
            _window.KeyUp += new System.Windows.Input.KeyEventHandler(_window_KeyUp);
            _window.Closed += new EventHandler(_window_Closed);
        }

        public void AttachSink(IMainWindowEvents sink)
        {
            _sink = sink;
        }

        // MainWindow events
        void _window_Closed(object sender, EventArgs e)
        {
            _sink.OnClose();
        }

        void _window_KeyUp(object sender, System.Windows.Input.KeyEventArgs e)
        {
            _sink.OnKey(e.Key.ToString());
        }

        void _goButton_Click(object sender, System.Windows.RoutedEventArgs e)
        {
            _sink.OnGoButtonClick();
        }

        // IMainWindow
        public void EnableGoButton(bool enable)
        {
            _window.Dispatcher.Invoke((Action)(() =>
            {
                _goButton.IsEnabled = enable;
            }));
        }

        public void SetThoughtText(string thought)
        {
            _window.Dispatcher.Invoke((Action)(() =>
            {
                _window.thoughtBox.Text = thought;
            }));
        }

        public void UpdteProgressBar(int percent)
        {
            _window.Dispatcher.Invoke((Action)(() =>
            {
                var min = _window.progressBar.Minimum;
                var max = _window.progressBar.Maximum;
                var range = max - min;
                _window.progressBar.Value = min + (percent / 100.0) * range;
            }));
        }
    }
}

Testing Cross-Platform Systems

This section discusses systems that run on different operating systems with identical behavior.

I've written a lot of cross-platform code, mostly in C++. Very often, nasty cross-platform problems turn out to be build issues (wrong flags, wrong dependencies, wrong version, etc). The key for cross-platform programming is to minimize platform-specific code as much as possible. This can be largely accomplished by using cross-platform libraries and avoiding direct system calls. With the possible exception of the user interface (which I'll discuss next), it is possible to develop entire cross-platform systems with a single code base. When that's not possible, you should strive to wrap any platform-specific code. The end result of such a design is that, by and large, the code you write is platform-agnostic and you have small, well-isolated pockets of platform-specific code that present a uniform interface to the rest of your code. This makes testing much easier. Developers can run tests on their development systems and, unless they touched some platform-specific code, they can be pretty sure that their changes will work on other systems.

Your build system should take care of testing on all platforms. This testing depends on your development process. Some teams run every test on every platform for every change. Some teams have layered testing policies where more expansive and exhaustive tests are run periodically (nightly or over the weekend, for example).

Some systems have to target platforms with different capabilities. It could be multiple hardware devices, different browsers, different versions of the same operating system/browser, etc. This situation can often lead to a combinatorial explosion of target platforms. You need to address it (as usual) with good design first. A common approach is to create flexible systems where capabilities are discovered dynamically or configured and integrated into the main system as plugins. From a testing point of view, this approach may reduce the testing load, too. If you have five optional capabilities, and your target platforms can support any subset of these capabilities, then you have 32 targets to test. But, if you can test each capability separately, then you only need to test five. This is an important distinction.

Another approach that can alleviate the burden is to simulate/emulate missing capabilities. For example, in the world of 3D graphics, software rendering is very common when hardware-acceleration is not available. From a testing point of view, this can be a double-edged sword. The code itself may become more streamlined and just assume a capability is available, not caring whether it's simulated or not. But, you will need to test the system behavior in both actual and simulated modes. This is especially true if you develop the simulator.

Testing the User Interface

Ah, testing the user interface…the bane of automated testing. There are several aspects of any user interface that need to be tested. In most cases, you will use some library that knows how to display widgets like buttons, text boxes, and images on the screen and allows you to hook up event handlers to events like button clicks or selection changes. If you develop your own custom UI, it is a different story and falls outside of the scope of this article. Directly testing a UI by trying to automate it is a nightmare. You have to deal with message loops, event queues, timers, input device latency,as well as take into account screen resolution and user response time. It can be done, but it's tedious and very brittle. A better approach isolates the display and low-level event handling from the management of the UI state. This approach uses an evolution of the famous MVC pattern called "Model View Presenter" (MVP). The models are the domain objects, which are completely agnostic of the UI. The views are a thin layer around the UI widgets (and compositions, such as windows and dialog boxes with a bunch of widgets). The presenter is the smart guy. It talks to model/domain objects and is responsible for updating the proper view when the model state changes. In addition, it also listens to events from the views, and manages the UI state. What does it buy you from a testing perspective? A complete freedom to test in isolation both your domain objects and your user interface logic and state. You rely on your widgets library to do the right thing and then validate the visual design manually.

The various class interfaces and their relationships are displayed in Figure 2.


Figure 2.

In the mind reader application, the domain or model is implemented by the TrivialMindReader class, which exposes the IMindReader interface:

    public interface IMindReader
    {
        void Read();
    }

It also generates mind reading events through the outgoing IMindReaderEvents interface:

    public interface IMindReaderEvents
    {
        void OnProgress(int percent);
        void OnReadComplete(string thought);
    }    


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video