Test runner for T4 unit tests


I noticed a clear trend of increasing complexity in the T4 text templates I wrote over the last few months. For simple templates, especially those that generate C# code, simply compiling the generated code is good enough to test it. But as the complexity of my templates gradually increased, I find myself having to spend a lot of time testing them manually. After almost a decade of using automated unit testing, manual testing is quite frustrating.

Many of the templates I create have to run in the Visual Studio T4 host in order to access Visual Studio extensibility APIs to perform advanced tasks, such as generating multiple output files from a single template. This makes it impossible to use the approach described by Jose Escrich which relies on an external (GAX) host for testing. Testing the templates by spinning up another instance of Visual Studio would painfully slow and complicated. This was the main reason why I had to test my templates manually.

Solution

The solution I came up with is a test runner that runs during template transformation, inside of the Visual Studio instance which is running the template under test. For example, suppose you have the following T4 template (SampleTemplate.tt).

<#+
    void GenerateFoo(string bar)
    {
#>
This is my template parameter: <#= bar #>
<#+
    }
#>

Here is what a unit test template might look like:

<#@ template debug=True#>
<#@ include file=UnitTesting.tt#>
<#@ include file=SampleTemplate.tt#>
<#
  RunTests();
#>
<#+
  [TestMethod]
  public void Test_GenerateFoo()
  {
    string parameterValue = “My Parameter Value”;
    GenerateFoo(parameterValue);
    Assert.IsTrue(GenerationEnvironment.ToString().Contains(parameterValue));
  } 

  [TestMethod]
  [ExpectedException(typeof(ArgumentNullException))]
  public void Test_GenerateFoo_ArgumentNullException()
  {
    GenerateFoo(null);
  }
#>

This template contains two methods marked with TestMethodAttribute defined by the Visual Studio unit testing framework. The RunTests method is defined in the UnitTesting.tt file, which you can download using the link at the bottom of this post. RunTests uses Reflection to find all methods marked with TestMethodAttribute and execute them. Inside these tests, you can use any of the standard Assert methods. TestInitializeAttribute, TestCleanupAttribute and ExpectedExceptionAttribute are supported as well.

The runner reports test execution results via Errors list of the generated TextTransformation, which is displayed in the Error List of Visual Studio. In the example above, both tests are successful and don’t produce any errors or warnings. Below is an example of a failed test and an inconclusive test and the output they generate.

<#@ template debug=True#>
<#@ include file=UnitTesting.tt#>
<#
  RunTests();
#>
<#+
  [TestMethod]
  public void Test_Inconclusive()
  {
    Assert.Inconclusive(”Sample inconclusive test”);
  } 

  [TestMethod]
  public void Test_Fail()
  {
    Assert.Fail(”Sample failed test”);
  }
#>

Unit Test Results

Note that errors and warnings indicate the file name and line number where failed tests are defined. Double-clicking a test error will take you to the source code of the failed test. This is handy when you have several templates with unit tests. In order for test runner to produce file name and line number you need to specify debug=true parameter in the <#@ template #> directive of the file that runs your unit tests (the template which calls RunTests).

Final Thoughts

Having the test runner running in the same process with template under test allows me to create (T4) unit tests for template code that uses Visual Studio extensibility APIs. T4 unit tests complement but don’t replace the traditional (Visual Studio) unit tests I create to test the actual generated code. Although it is possible to unit test actual generated code in T4 unit tests using dynamic compilation, as illustrated in Jose’s article, it requires heavy use of Reflection, which I have not been able to justify for templates I worked on so far.

Reporting of results in the T4 test runner could be improved. In particular, it would be nice to generate Message items in the Error list to indicate which tests ran successfully. Unfortunately, this requires use of Visual Studio extensibility APIs which would make test runner more complex than my time allows. It would also be nice to provide detailed exception information, similar to Visual Studio test results.

Download

Source Code


Write a Comment

Take a moment to comment and tell us what you think. Some basic HTML is allowed for formatting.

Reader Comments

Be the first to leave a comment!