Microsoft Pex Framework and its new cousin Code Digger

Unit Testing has been around for a long time and today it has become an important aspect of Software Development. Unit Testing has evolved over the years and there has been significant research into improving both tooling and techniques around Unit Testing. Microsoft Pex is one of the research tools that is slowly but steadily getting popular within .NET world. Pex stands for “Program Exploration” and it makes Unit Testing lot easier and fun than ever before.

Microsoft Pex has adapted the concept of White Box testing, which is basically testing an application or function based on its internal working structure. The internal structure can be categorized in number of ways such as Branches, Paths, Data flow etc.

This article is published from the DotNetCurry .NET Magazine – A Free High Quality Digital Magazine for .NET professionals published once every two months. Subscribe to this eMagazine for Free

Getting started with Microsoft Pex

You can download Pex from Microsoft Research Pex website. Note that Pex also comes with a tool called Moles, which is now replaced by the powerful VS 2012 Fakes framework.

Pex can be used from VS2005 onwards. However the full VS integration is only available from VS2008. The Pex examples that have been demonstrated in this article use VS 2010.

At the time of writing Pex is not supported by the new VS 2012. It has been replaced by a tool called Code Digger which is a lightweight version of Pex. I will describe Code Digger in more detail in a bit. The main idea behind Pex is based on Parameterized Unit Testing. We will take a look at this in more details in the section “Parameterized Unit Testing with Pex”. For now, it is important to understand that Pex helps us to auto generate Unit Test cases based on the structure of the program. This means that you no longer need to write hand written Unit Tests to extensively exercise a method, as Pex framework is capable of auto generating a significant set of test cases for you.

Does Pex completely replace hand written Unit Tests?

The short answer is No. In practice, I find that Pex is really good for generating various test cases for functions that are predominantly based on many branches that can take various range inputs as parameters. A good example would be an algorithm that you want to extensively test various inputs in. For these types of algorithms, hand written Unit Tests are exhaustive to write and have no guarantee that we cover all the required test cases.

In most other cases, you still require hand written Unit Tests. For example, you want to be sure that your components are well designed as you would normally do with practices such as TDD (Test Driven Development).

Once you have installed Pex in VS 2010, you can see the Pex commands in the context menu as below.

Generating Unit Tests with Pex

Let’s have a look at a C# function, and generate the test case for this function using Pex. We have the following sample function to calculate Salary

public class Employee
{
public decimal CalculateSalary(int experience)
{           
  if (experience > 20)
  {
   throw new ArgumentException("Experience cannot be greater than 20");
  }
  decimal baseSalary = 1000.00m;
  if (experience < 5)
  {
   return baseSalary / experience;
  }
  if (experience >= 5 && experience <= 10)
  {
   return baseSalary + 500.00m;
  }
  if (experience > 10)
  {
   return baseSalary + 1000.00m;
  }
  return baseSalary;
}
}

In the menu after clicking “Run Pex”, you can see Pex has generated the inputs and outputs as below.

Pex also generates inputs that can potentially make the program to throw unhandled exceptions, such as DivideByZeroException. The good thing about this is that you can look at the failing test and modify your program, so it can handle DivideByZero exceptions.

public decimal CalculateSalary(int experience)
{
if (experience > 20)
  throw new ArgumentException("Experince cannot be greater then 20");
decimal baseSalary = 1000.00m;
if (experience < 5 && experience > 0)
  return baseSalary / experience;
if (experience >= 5 && experience <= 10)
  return baseSalary + 500.00m;
if (experience > 10)
  return baseSalary + 1000.00m;
return baseSalary;
}

How Pex generates inputs and outputs

The Pex engine looks at the method’s every branch, statement, and data. It then generates inputs and outputs accordingly. The technique called Dynamic Symbolic Execution is used to create these inputs and outputs. As you see, Pex feeds the simplest possible values first to a method and incrementally it generates inputs for corner cases.

If you were to add another condition, Pex would generate another entry in the table. This means that you would get really high code coverage for your routine.

Pex generates all these Unit Tests separately in a code generated file which you have access to. See some examples below.

[TestMethod]
[PexGeneratedBy(typeof(EmployeeTest))]
[ExpectedException(typeof(ArgumentException))]
public void CalculateSalaryThrowsArgumentException38()
{
decimal d;
Employee s0 = new Employee();
d = this.CalculateSalary(s0, 21);
}

[TestMethod]
[PexGeneratedBy(typeof(EmployeeTest))]
public void CalculateSalary94()
{
decimal d;
Employee s0 = new Employee();
d = this.CalculateSalary(s0, 20);
Assert.AreEqual(200000e-2M, d);
Assert.IsNotNull((object)s0);
}

This also means that if you cannot understand any input and how the code produces a corresponding output/result, you can easily debug the generated result. (Right click on any test and select “Debug”).

All Pex generated Unit Tests can also be run separately from standard VS test tool window.

Each of the above generated test calls a Pex parameterized test method as below.

///

Does Pex support other test frameworks?

Pex does not support other test frameworks out of the box. However you can download Pex Extension from Codeplex, that allows you to use Pex with your own test framework

Parameterized Unit Testing with Pex

One of the neat features of Pex is that you can create parameterized Unit Tests to customize your unit Tests. Once you specify a parameter in your Unit Test method as argument, Pex will auto generate inputs (based on the branches and the statements), call the system under test and then assert the return value. To make parameterized Unit Test, you must decorate the Unit Test method with [PexMethod] attribute.

[PexMethod]
public void CalculateSalary(int experience)
{
var sut = new Employee();
var result = sut.CalculateSalary(experience);
Assert.AreEqual(1000.00m, result);
}

Note that generating Parameterized tests can also be achieved by right clicking the class under test, selecting Pex > Create ParamaterizedUnitTests. Once you create your own parameterized test method, right click on the test method and select “Run Pex Explorations”. You should be able to see the Pex generated inputs and outputs Assert against the Unit Test’s assert condition.

If you want additional customization against the parameters that has been passed into the Unit Test, you can use the [PexArguments] method as below. This will generate additional inputs and outputs based on the arguments specified by the [PexArguments] attribute.

[PexMethod]
[PexArguments(-2)]
public void CalcualteSalary(int experience)
{
var sut = new Employee();
var result = sut.CalculateSalary(experience);
Assert.AreEqual(1000.00m, result);
}

You can easily debug any failing tests using standard Visual Studio debug features.

Microsoft Code Digger

Code Digger is a lightweight version of Pex that specifically targets VS 2012 Portable class libraries. At this stage, Microsoft Pex, which we looked at, is not compatible with VS 2012 yet. I believe the future version of Code Digger may replace Pex and would target other project types.

You can download the Code Digger Extension from VS Extension Gallery

Once you have installed the Code Digger plugin (VS Professional and above), you should be able to see a new menu item “Generate Inputs/Outputs Table” in the context menu.

Let’s say we have a simple C# method that checks specific names within a list.

public bool ListContainNames(List listNames)
{
if (listNames == null) return false;
if (listNames.Any())
{
  if (listNames.ElementAt(0).Contains("smith"))
  {
   throw new Exception("should not contain smith");
  }
  if (listNames.ElementAt(0).Contains("david"))
  {
   return true;
  }
}
return false;
}

After running Code Digger, you should be able to see a table that generates inputs and outputs very similar to Pex generated table which we saw earlier.

Note: Unlike Pex, at this stage Code Digger does not have any auto generated Unit Tests or parameterized Unit Tests.

Pex for Fun (Pex4fun.com)

What we saw so far is automatic test case generation integrated within Visual Studio. Pex4Fun is web site that uses the same Pex engine in the server side but provides more interactive or gaming style experience to the user. It is a simple interface, and you can post C# code in the text editor and click “Ask Pex”. Pex sends the code to the server and validates the code, generates test data and sends back the result as follows

If you copy and paste this code in Visual Studio, you would see pretty much the same test cases generated by Visual Studio PEX engine.

The idea of this web site is to provide puzzles and Coding Duels so it can gamify Unit Testing. Once you have selected a puzzle, and after generating inputs and outputs, your task is to modify the program so the Pex no longer produces any failed results. Once you have no failed results, you have solved the puzzle. It is fun and you can use this program to solve various puzzles.

Summary

As you know these are research tools and they have not yet made it into the final/latest Visual Studio. If you are using VS 2010, you can use Pex extensively and generate a boiler plate regression test suite. Code Digger seems to be using the same Pex engine for test generation and we would expect substantial changes around automated testing in upcoming VS releases. It is very promising and lot developers would benefit from.

I recommend use Pex for complex algorithms, to get a leg up in generating maximum possible test cases automatically. Once Pex is done, we can review the generated test cases and augment them with hand-written Unit Tests. There are still great benefits of writing your hand written Unit Tests (TDD or Non-TDD).

Pex4Fun is a great web site that allows you to solve coding puzzles.

Downloaded the entire source code of this article (GitHub)

This article has been editorially reviewed by Suprotim Agarwal.

C# and .NET have been around for a very long time, but their constant growth means there’s always more to learn.

We at DotNetCurry are very excited to announce The Absolutely Awesome Book on C# and .NET. This is a 500 pages concise technical eBook available in PDF, ePub (iPad), and Mobi (Kindle).

Organized around concepts, this Book aims to provide a concise, yet solid foundation in C# and .NET, covering C# 6.0, C# 7.0 and .NET Core, with chapters on the latest .NET Core 3.0, .NET Standard and C# 8.0 (final release) too. Use these concepts to deepen your existing knowledge of C# and .NET, to have a solid grasp of the latest in C# and .NET OR to crack your next .NET Interview.

Click here to Explore the Table of Contents or Download Sample Chapters!

Author

Raj Aththanayake is a Microsoft ASP.NET Web Developer specializing in Agile Development practices such Test Driven Development (TDD) and Unit Testing. He is also passionate in technologies such as ASP.NET MVC. He regularly presents at community user groups and conferences. Raj also writes articles in his blog http://blog.rajsoftware.com. You can follow Raj on twitter @raj_kba