Channels ▼
RSS

C/C++

Testing Complex C++ Systems


In part one of this three-part series on deep testing complex systems, I covered the practical aspects of deep testing and demonstrated how to test difficult areas of code, such as the user interface, networking, and asynchronous code.

More Insights

White Papers

More >>

Reports

More >>

Webcasts

More >>

In this article, I discuss some techniques and utilities I have used to successfully test complex systems in C++. In the final article next week, I'll discuss similar complex testing for Python (with Swig C++/Python bindings) and high-volume, highly available C#/.NET Web services.

Obstacles to Testing Complex C++ Code

C++ arguably poses the most difficult challenges for testing among the major industrial-strength programming languages and environments. There are several challenges to deal with: Build speed is slow; the language uses weak types; and C++ programmers love twisting code or template-based designs to extract every last bit of performance. Let's look at how these factors affect testing complex code.

Slow Build Speeds

C++ projects (especially cross-platform projects) are often built as one big monolith. They can comprise large numbers of static project libraries along with many more third-party libraries that are all linked together to generate the final executable (often, several executables).

The result of this monolithic construction is less than stellar dependency management. I've seen several large C++ codebases where every merge was a project in itself involving a dedicated build engineer to convene multiple developers to help resolve conflicts and build the software, sometimes requiring multiple building passes to combat cyclic dependencies. Add the tools used to build many cross-platform C++ projects (such as make) and you get a picture of slow, very complex builds.

The impact of this on testing is profound. When simply building the software is such a chore, the tendency is to minimize the number of artifacts you generate. This leads to one of several unpleasant choices:

  1. No tests — just build the software and play with it; trust your QA team to find all the bugs.
  2. Add some test support to the main software with compile-time or runtime switches.
  3. One big test executable (or test framework) that links against all the dependencies and contains all the tests.
  4. Many small test executables, each linked against its dependencies.

Option #1 is an obvious no-go from serious testing point of view (but very common in practice).

Option #2 is cumbersome. It doesn't let you to perform isolated testing and complicates the final product with testing code and dependencies.

Option #3 requires a slow build of the huge test executable whenever you make a small change (either to the code under test or to the test itself).

Option #4 requires a lot of tinkering around. It's pretty good when you work on an isolated bug or feature and has to build just one small test executable, but when you want to run a suite of regression tests, you will have to wait a long time for the build because each and every small executable will have to link against all its dependencies, which will duplicate a lot of effort (especially for common libraries used by every test executable).

The Quest for Performance

C++ was designed for raw performance. You pay only for what you use. It is compatible with C, which started as a glorified assembly language. In later years, C++ became a multi-paradigm language supporting procedural, object-oriented, and generic programming styles, while still maintaining its C compatibility and performance. However, this led to a programming language that is an order of magnitude more difficult to learn and practice than other mainstream languages. In its glory days (early '90s), C++ was used for everything, especially on Windows. You wrote your number-crunching code and your user interface and your networking code in C++, and that was that. The rise of the Web and Java changed everything. Suddenly, the programming language of your project wasn't a given and you could even mix and match programming languages in the same project. Later, hardware got faster, other languages got better (C#, Python), and Web applications where developed in every possible language except C++ (anyone remember ATL server?). That relegated C++ to the engine to do what it does best — fast processing. C++ programmers (unless they are polyglot and dabble in other languages) justifiably focus more than anything on generating fast and tight code. Many great C++ developers formed their habits before automated unit testing and the agile movement made it to the big stage. Also, because performance is very often a system-wide property and not just the sum of the performance of the various components, its pursuit leads to a dependence on the usefulness of unit testing.

Very Weak Type System

C++ is a statically typed language, but its type system is very weak. You can freely cast any pointer to any other pointer using reinterpret_cast. It can perform implicit casts/coercions/promotions on your behalf in certain situations. In addition, it has the legacy C preprocessor that allows inventing private sub-languages in a non-typesafe way. Add to that powerful templates with no constraints. Because all these features are heavily used in industrial strength programs, you end up with a combustible concoction that is pretty difficult to reason through and to feel secure that you have covered in your tests all the ways the code can misbehave.


Related Reading






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Comments:

AndrewBinstock
2013-02-28T20:08:13

You're quite right. Sorry about that. We should have broken that up a bit during the editing process.


Permalink
ubm_techweb_disqus_sso_-56bc6f0e638befb69975ad60ad27455b
2013-02-28T19:48:53

I wish the style of the article was more reader friendly somehow.
The sentence as below is a brain hammer.

"I created plugins that reused the logic inside the fixed algorithm nodes and then constructed experiments that used the same networks with the same algorithm types and connectivity except that on one network the nodes used were the concrete (old) sub-classes of the runtime engine, and on another network all the nodes were PluginNode hosting the same algorithmic code"


Permalink
rudmerriam
2013-02-07T22:32:59

A quick web search indicates that strong vs weak typing is neither well defined nor objective.


Permalink
Zenju
2013-02-07T11:44:26

> it's just a technical term with a well-defined and objective meaning

Now I'm interested where you got your "sources" from. Or did you just make this up?


Permalink
Zenju
2013-02-07T11:36:53

> asked before speaking
And in order to "ask" I would "speak" I guess?

> what you *can* do with the language

That's not true. In this vague sense you can do anything with every language. C++ is commonly seen as a strongly typed language. There is no discussion about it.


Permalink
ubm_techweb_disqus_sso_-03ddef15ee57994d998f6ecaadbe57fe
2013-02-07T06:46:17

A language is weakly or strongly typed depending on what you can do with the language, not what uses are considered bad. It has no pejorative connotation; it's just a technical term with a well-defined and objective meaning. C++ is weakly typed.


Permalink
ubm_techweb_disqus_sso_-03ddef15ee57994d998f6ecaadbe57fe
2013-02-07T06:43:16

You should have asked before speaking. The weakness or strength of the type system of a programming language is a technical concept, without any pejorative connotation. It doesn't mean bad (nor good), and talks about what you *can* do with the language (not what you *should* do).


Permalink
AndrewBinstock
2013-02-06T20:40:33

I agree with most of your points. The terminology has taken on a bit of a political aspect within the C++ community. There is no doubt that by using the modern features in C++ with discipline, type safety can be assured. The "weak" part (an unfortunate descriptor, perhaps mutable and fixed would have been more neutral terms) comes in large part, but not exclusively, from the legacy of C. In this article, the author is discussing testing legacy codebases, so in that context, the challenges faced by testers due to the use of weak typing features seems appropriate to me. Zenju is right in his implication that developers who use the features are doing the linguistic equivalent of swearing. But the author's discussion of this is certainly not an attack on the language. It's simply a reflection of the reality of legacy codebases.


Permalink
ubm_techweb_disqus_sso_-4a1cc78416705fdb5465e5351fbffb09
2013-02-04T20:23:08

C++ is not weakly typed by most people's definition of what weakly typed means: that all class instances would be stored in an "object" type at compile-time, like C#'s ArrayList elements. On the other hand, a strongly typed language, by most people's definition, would have the types written out explicitly in the code; you could then perform compile-time error checking. The analogous C# collection in my example would be List<int>. C++ has always been a strongly typed language.

I think there has been an (unfortunate) conflation in recent years of weakly typed typed systems and type systems which happen to not have excellent security guarantees. Programming terms, jargon, motivations, concerns, etc. don't revolve around a type system's security guarantees; it's usually not the most important thing. I think many of us know that this specific denigration of the C++ language is basically propaganda by people who are promoting other languages with big security features.


Permalink
AndrewBinstock
2013-02-04T06:39:34

Zenju: I can't tell if you're trolling or not, but 1) C++ is weakly typed. Do you really need a link for this? 2) C++ does take a long time to build. The article provides a link to Walter Bright's explanation of why C++ compiles slowly. Bright has written three commercial C++ compilers.3) Fast tight code is precisely the reason most C++ developers I know use the language. And given my line of work, I speak to a lot of C++ developers.

I'm not quite sure what you're so upset about. The author's claims seem reasonable to me and not inconsistent with my experience.


Permalink
Zenju
2013-02-03T11:30:01

I do not see where the author makes a distinction between the language and code. Quite contrary he mixes it all together producing a number of statements that are wrong in their generality:

"C++ is a statically typed language, but its type system is very weak."
"Build speed is slow"
"C++ arguably poses the most difficult challenges for testing"
"C++ programmers love twisting code"
"C++ programmers [...] justifiably focus more than anything on generating fast and tight code"

By making these dubious statements without any reasoning right at the beginning of the article the author loses credibility and the reader interest to go on.


Permalink
rudmerriam
2013-01-31T21:30:19

I'll support Zenju's comment. Saying C++ is weakly typed - regardless of the context, i.e. legacy systems - because a developer misused reinterpret_cast is a very misleading statement. This is an abuse of the statement, not the fault of the language. The casting operators are there for a purpose and useful when appropriate. They were introduced to provide control over the casting of C specifically to keep C++ a strongly typed system.

Regardless of how large the system is it might behoove the company to grep for reinterpret_casts and fix them.


Permalink
AndrewBinstock
2013-01-30T22:58:55

Not sure I understand your comment or your recommendation. It's perfectly clear that the author is discussing code bases that already exist.


Permalink
dblake950
2013-01-30T15:09:22

Of course, you are right about C++. But the author is pointing out some of the common problems you run up against when prepping tests on large, complicated systems -- and naughty usage is certainly common enough.


Permalink
Zenju
2013-01-30T11:14:53

> type system is very weak. You can freely cast any pointer to any other pointer using reinterpret_cast.

"the english language is bad - you can swear a lot in it!". The author doesn't seem to be able to distinguish between what you can and what you should do. It's the common fallacy of confusing the code of some bad C++ programmers with the C++ language in general. I would suggest he read some new books on how C++ is used today, especially C++11.


Permalink

Video