Channels ▼


Testing Complex Systems

The first rule is to isolate your system code from the low-level networking. This is very similar to the MVP pattern for the user interface. You want your domain objects to be implemented in a network-agnostic fashion. But, you have to be careful when you design your interfaces, because chatty or blocking interfaces won't cut it when you have to send a lot of data over the wire. In general, for high-performance networking, you would want asynchronous interfaces. I'll talk more about asynchronous code later.

To demonstrate the process, I created a little mind reading TCP server. The protocol is very simple. The server waits for a client to send the command: "Read thought remotely." The server responds with a progress report in the format "progress:<percent>" (for example. "progress:25"), and finally, it sends the thought itself as thought: <thought> (for example, "thought:I think therefore I am"). The server is not very interesting, but it can handle multiple concurrent clients. I also created a Client class, which exposes the IMindReader interface and sends IMindReaderEvents to a sink (see Listing Five) .

Listing Five

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net.Sockets;
using System.Threading;
using System.IO;

namespace MindReader.Client
    public class Client : IMindReader
        string _hostname;
        int _port;
        IMindReaderEvents _sink;

        public Client(string hostname, int port)
            _hostname = hostname;
            _port = port;

        public void AttachSink(IMindReaderEvents sink)
            _sink = _sink;

        public void Read()
            new Thread(() =>
                var client = new TcpClient();
                client.Connect(_hostname, _port);

                // Wait for connection
                while (!client.Connected)

                var s = client.GetStream();

                // Send read command to server               
                var bytes = ASCIIEncoding.ASCII.GetBytes("Read Thought Remotely\r\n");
                s.Write(bytes, 0, bytes.Length);

                // Read progress reports and finally the thought
                var reader = new StreamReader(s);
                while (true)
                    var line = reader.ReadLine();
                    if (line.StartsWith("progress:"))
                        var progress = int.Parse(line.Split(':')[1]);

                    if (line.StartsWith("thought:"))
                        var thought = line.Split(':')[1];


This Client class is a drop-in replacement for the local mind reader and requires a single line change in the factory to hook all the components together:

            var mainWindow = new MainWindow();
            var wrapper = new MainWindowWrapper(mainWindow);

            //var mindReader = new TrivialMindReader();
            var mindReader = new cli.Client("localhost", 54321);

            var presenter = new Presenter(wrapper, mindReader);


The IMindReader and IMindReaderEvents were designed for asynchronous interactions initially, so this is a very easy transition. The Client runs on a separate thread and uses blocking APIs to interact with the server. In this example, I created two simple test cases to verify the entire session. I have a simple Sink class that implements the IMindReaderEvents and collects the calls (similar to the MockWindow in the PresenterTest):

    class Sink : IMindReaderEvents
        public List<int> OnProgressCalls = new List<int>();
        public List<string> OnReadCompleteCalls = new List<string>();

        public void  OnProgress(int percent)

        public void  OnReadComplete(string thought)

I pass Sink objects to the Client and call its Read() method to start a mind reading session against the server:

        public void SingleClient_Test()
            var cli = new Client(_hostname, _port);
            while (_sink.OnReadCompleteCalls.Count == 0)
            Assert.AreEqual(3, _sink.OnProgressCalls.Count);
            Assert.AreEqual("I think therefore I am", _sink.OnReadCompleteCalls[0]);

I also created a test for two concurrent clients:

        public void TwoConcurrentClients_Test()
            var sink1 = new Sink();
            var sink2 = new Sink();

            var cli1 = new Client(_hostname, _port);
            var cli2 = new Client(_hostname, _port);

            // Start 2 mind reading sessions
            // Wait until both clients got a reading
            while (sink1.OnReadCompleteCalls.Count * sink2.OnReadCompleteCalls.Count == 0)
            Assert.AreEqual(3, sink1.OnProgressCalls.Count);
            Assert.AreEqual(3, sink2.OnProgressCalls.Count);
            Assert.AreEqual("I think therefore I am", sink1.OnReadCompleteCalls[0]);
            Assert.AreEqual("I think therefore I am", sink2.OnReadCompleteCalls[0]);

Mock Servers and Clients

In networking code, it is common to talk about clients and servers. There are many topologies and protocols, but generally, clients are configured or know how to find their server/s and initiate communication. In the ideal testing scenario, you completely isolate your code via interfaces. If your code is a server, you create a mock client and, in the test, your mock client will start talking to your server. If your code is a client, you create a mock server and, in the test, your code will start talking to the mock server. Some times the same code is both a server to some clients and a client of other servers/services. In such cases, you will provide multiple mocks and test the server roles and the client roles separately.

The fun thing about testing against a mock server or client is that you have complete control and can easily simulate any failures, such as an unreachable server, sudden disconnect at any point, missing responses, corrupt responses, and slow responses. You can even insert breakpoints and step through each interaction in your code and in the mock. When testing against real servers/clients, it is often very difficult to create these many error conditions and behaviors. Mock servers are also great for emulating clusters, where you dynamically reconfigure your mock cluster on the fly from test to test or even in the middle of a test.

The MindReader client and server repeat the same concepts precisely. Instead of working directly with the .NET TcpListener and TcpClient, you would create ITcpListener, ITcpListenerEvents, and ITcpClient interfaces (if you want a non-blocking client, then add ITcpClientEvents). Then add a TcpListenerWrapper and TcpClientWrapper, hook everything up, and pass it to the MindReader client and server. From there, you would be in a position to simulate any TCP networking scenario.

Localhost Test Servers

Sometimes mocking a server or a client faithfully is complicated. In these cases, the next best thing is a local test server (or client). You don't have full visibility into all the interactions (usually only via log files), but at least the connection is greatly simplified. You can start and kill test servers on your machine without impacting anyone else. Your code also interacts with a real server, which gives you confidence that it will work in the wild, too. The debugging experience is usually degraded in such cases.

When testing the MindReader networking scenario, I ran the server locally. All it takes from the client side is to make sure the connection information is not hard-coded:

    public class ClientServerTest
        Sink _sink;
        string _hostname = "localhost";
        int _port = 54321;

        public void SingleClient_Test()
            var cli = new Client(_hostname, _port);

Dedicated Test Servers and Environments

Local servers can be hard to configure (for example, when the server depends on many other services that need to be mocked or configured). Often, you will need to test your code against multiple servers and/or clients. This is where dedicated test servers or full-fledged test environments come into play. Test environments try to mimic the production environment to some degree. There are two ways to work against a test environment:

  1. Run your code locally and connect to the test environment. This option is usually good for testing client code.
  2. Deploy your code to the test environment. Test via the exposed API (REST, SOAP, plain HTTP, custom TCP/UDP). This option is usually good for testing server code.

Test environments are useful for integration testing. The main thing to pay attention to is that the test environment doesn't deviate from the production environment: You must have a process in place (ideally automated). The other concern is version control of different components and resources under test. Test environments are a scarce resource shared by many developers. If multiple developers deploy their latest code at the same time, they can easily step on each other's toes and break the test environment for everyone.

Latency, Throughput, and Stress Testing

When dealing with distributed systems, two crucial metrics are throughput and latency. Throughput defines how much data can be transferred over the wire, and latency defines how fast can you respond to a request. Due to the nature of distributed systems, these attributes depend on many factors, such as the hardware specifications of different nodes, the bandwidth available between nodes, the protocols in use, and the load on your system. There is also often an interplay between latency and throughput. In general, when the load on the system increases the latency will start to go up, and the throughput may degrade, too. It is very difficult, if not impossible, to predict the behavior of a distributed system under heavy load because it is non-linear. Various thresholds and interactions kick in at different loads. Knowing how much load your system can handle is crucial to ensuring quality of service and scalability.

The way to acquire this knowledge to test the system under various simulated loads and observe its behavior. Sometimes, you isolate different components of the system and stress test them independently. There are many tools available for stress testing, but for sophisticated systems, you would want to create your own stress tool, which can simulate realistic loads and patterns of interactions.

Limited Production Testing and Gradual Deployment

Stress/load testing is important, but it is not enough. Another good technique for testing code changes is to deploy the new version of the code to a small number of servers side-by-side with the existing system and observe the behavior of the new servers. If anything goes wrong, you switch back to the existing version and analyze the problem. If everything runs OK, you can transition more and more servers to the new system. This gradual deployment is an effective technique, but is a complicated process with multiple steps, especially if code changes involve database schema, protocol, or file formats.

Testing Persistent Storage

Persistent storage is another area that is not always easy to test. Most likely, you will not be able to run tests against actual production storage (especially not tests that modify the data). This means that you will have to test against test data, which comes with its own challenges involving synchronization with production data. There are two levels here:

  1. The DB schema or file formats used for persistence
  2. The actual content

When you keep persistent test data, each level (or both) may get out of sync. You have to be very diligent to stay ahead of the curve. The first line of attack is as usual — abstracting away data access and testing your code against mock data sources. This will enable running lots of tests quickly without actually accessing the real data store. In some cases, you may need to test more sophisticated data access scenarios involving caching at different levels. The key is to be able to control every aspect of the data access process that you want to test. As long as you interact with dependencies through interfaces and use dependency injection, you're good to go.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.