Channels ▼
RSS

Design

Pex: Microsoft Research's Unit Test Generator and Evaluator


Pex is a testing tool developed by Microsoft Research. It provides three very useful capabilities to support testing. First, it can explore your code and suggest which tests you should have. Second, if you have parameterized tests, Pex can figure out which combination of parameters needs be tested in order to give you full coverage of possible scenarios. And finally, if you use Code Contracts, then Pex will employ that information to fine-tune the unit tests it suggests or generates for you. I'll review the three aspects in more detail after a quick look at installation and set up.

You can download Pex from Microsoft. If you're curious about the underlying technology, you can also access a few PDF documents explaining the internal workings of the tool from the site. Pex is a standard add-in for Visual Studio and should be installed like any other.

Suggesting Unit Tests

Now let's look at what Pex can do. Suppose you are not a TDD proponent; at some point, you may happen to have a C# class and no unit tests for it. With Pex, all you need do is right-click on the class code in Visual Studio and click the Pex|Create Parameterized Unit Tests menu item. The effect is that a new test project is created and added to the selected solution; moreover, the test project can target MSTest — the default Visual Studio testing framework — as well as other frameworks, including NUnit. Open up the project and you'll find good parameterized tests are there for you to complete and run. However, auto-generated tests don't really do much except invoke a method with parameters. At the very minimum, you'll want to complete them with some assertions.

A parameterized unit test is simply a test that accepts parameters; parameters can be provided in a number of ways depending on the framework. Typically, you pass parameters from a database file. In Visual Studio MSTest, you need to write a wrapper parameter-less test configured to know about the data source. This wrapper test will be invoked iteratively; it reads values from the current row and passes those values to the parameterized test. The challenge is choosing meaningful input values for the tests. This is where the second key feature of Pex comes to the rescue.

Choosing Meaningful Input Values

Open the previously created unit test file and right-click on it. This time, click on the Run Pex Explorations menu item. At this point, Pex analyzes the code of the method under test and figures out which values it would make sense to test for. Pex uses an implementation of the dynamic symbolic execution technique that basically processes the code and calculates the results of the method execution. Every time a new possible result is found, Pex determines which combination of values leads to the result and adds the values to the list. In this way, it iteratively discovers all possible outcomes of a method and, at the same time, it also determines a way to produce them.

The result of a Pex exploration is a list of auto-generated classic unit tests, which call the method with any meaningful combination of inputs previously determined. You can just copy these tests into a test project and integrate them in the build process with the guarantee of a high coverage and, more importantly, strong insurance against corner cases. Generating parameterized unit tests is often a long and boring process — Pex automates that process and reduces it to just a click or two.

Code Contracts and Pex

Fully integrated in .NET Framework 4.0, Code Contracts are Microsoft's implementation of the tried-and-true idea of software contracts — most notably, preconditions, postconditions, and invariants. In particular, Code Contracts are used to annotate methods with conditions that must be preliminarily verified for the code to run. As an example of Code Contracts and Pex integration, consider the following scenario. Suppose you have a method that does some work on a string parameter. Suppose also that your implementation blindly assumes the received string is a non-null string. You are therefore potentially exposed to a null reference exception. A quick exploration by Pex reveals the issue, in the form of a notification, which allows you to add a test.

A warning about a possible defect is a great help, but Pex can do more. If Code Contracts are enabled for the project, then Pex will offer to add a proper precondition to your code so that you can catch any invalid data and take control of the situation. Pex will run an analysis of the code and build a base of knowledge about your code. Code Contracts provide additional information that contributes to an even more accurate analysis from Pex.

In summary, Pex is an innovative white-box testing tool that can be used both as an aid to generate nontrivial unit tests and as a peer reviewer to quickly look at your code and find holes and omissions in it. Pex is not dependent on Code Contracts, but it can use Code Contracts to enhance your tests if they are already employed in your code. One limitation I should mention is that it works only with managed code.

More Testing

Pex is an innovative testing tool that represented a deeper commitment to unit testing than Microsoft had previously exhibited. Since it was released, Microsoft has continued bringing out tools that facilitate unit testing. In Visual Studio 2012, for example, it shipped a framework called Microsoft Fakes. Fakes facilitates the construction of unit tests with stubs and mocks, plus a runtime redirection of code in what the company calls "shims." These technologies will be discussed in upcoming articles. Meanwhile, if you need help generating humdrum unit tests that exercise code thoroughly, download Pex and have it do most of the heavy lifting.

[For a hands-on look at using Pex, see "Working with Microsoft PEX Framework" — Ed.]


Dino Esposito is a frequent writer on Microsoft developer technologies.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video