T4 Tutorial: Unit testing code generators


This post is a part of the series that introduces code generation with Text Templates (also known as T4 Templates) in Visual Studio using C# and Visual Basic; explains how to create reusable templates and combine them in complex code generators. In order to follow examples in this article, you need to have Visual Studio 2008 Standard Edition or higher, SQL Server 2005 or later, T4 Toolbox and T4 Editor installed on your computer.

Introduction

As complexity of a code generator grows, testing becomes increasingly important to verify its correctness. It becomes even more important to ensure backward compatibility later on, when multiple teams start using the code generator on multiple projects. While it is possible to perform adequate amount of testing manually, few developers will find this activity exciting. Instead, automated unit testing became popular and widely adopted by developers over the past decade. It allows you to implement test scripts in the form of code and execute them to verify correctness of system under test (a code generator in this case) . Compared to manual testing, it requires more effort upfront to create the tests; however, execution of the tests is very quick, repeatable and doesn’t require developers to perform tedious manual tests over and over.

Automated testing of T4 code generators is challenging because the code generation engine compiles and executes them on-the-fly. This makes it difficult to use standard Visual Studio test projects for automated testing. To overcome this challenge, T4 Toolbox provides a test runner that can execute unit tests included as part of the code generator under test.

In the previous article of this series, we added error-handling logic to a code generator that produces CRUD stored procedures for all tables in a SQL database. Error handling logic is a great candidate for automated unit testing because it doesn’t get executed during normal, “happy” scenarios and requires an explicit effort to test. Without automated unit testing, the error-handling logic itself is likely to have errors that will go unnoticed until users find them.

C#
<#@ … #>
<#+
public class CrudProcedureGenerator : Generator
{
    // …
    protected override void Validate()
    {
        if (string.IsNullOrEmpty(this.DatabaseName))
        {
            this.Error(
                “DatabaseName property of CrudProcedureGenerator ” +
                “must be assigned”);
        }
    }
}
#>

Visual Basic
<#@ … #>
<#+
Public Class CrudProcedureGenerator
    Inherits Generator 

    ‘ …
    Protected Overrides Sub Validate()
        If String.IsNullOrEmpty(Me.DatabaseName) Then
            Me.Error( _
                “DatabaseName property of CrudProcedureGenerator ” & _
                “must be assigned”)
        End If
    End Sub
End Class
#>

Validate method (shown above) verifies that user provided a value for the DatabaseName property of the CrudProcedureGenerator and reports an Error otherwise. In the remainder of this article we will use this code as an example to talk about creating, running and troubleshooting unit tests for T4 code generators.

Create a new unit test

  • Open the the project which contains the CrudProcedureGenerator you created previously. This source code is also available for download at the bottom of this article.
  • Right-click the T4Tutorial project in the Solution Explorer and select Add->New Item from the context menu.
  • In the Add New Item dialog, select Code Generation->Unit Test template; enter CrudProcedureGeneratorTest.tt as the item name and click the Add button. If you don’t see the Code Generation folder, make sure you have installed the latest release of T4 Toolbox on your computer.

Add New Item dialog, Unit Test template

  • Double-click CrudProcedureGeneratorTest.tt in the Solution Explorer
C#
<#@ template language=”C#v3.5hostspecific=”Truedebug=”True” #>
<#@ output extension=”log” #>
<#@ include file=”T4Toolbox.tt” #>
<#@ include file=”T4Toolbox\UnitTesting.tt” #>
<#@ import namespace=”System.Diagnostics” #>
<#+
// <copyright file=”CrudProcedureGeneratorTest.tt” company=”Microsoft”>
//  Copyright © Microsoft. All Rights Reserved.
// </copyright> 

[TestClass]
public class CrudProcedureGeneratorTest
{
    [TestInitialize]
    public void TestInitialize()
    { 

    } 

    [TestCleanup]
    public void TestCleanup()
    { 

    } 

    [TestMethod]
    public void Test_Success()
    {
        Assert.Inconclusive();
    } 

    [TestMethod]
    [ExpectedException(typeof(Exception))]
    public void Test_Exception()
    {
        Assert.Inconclusive();
    }
}
#>

Visual Basic
<#@ template language=”VBv3.5hostspecific=”Truedebug=”True” #>
<#@ output extension=”log” #>
<#@ include file=”T4Toolbox.tt” #>
<#@ include file=”T4Toolbox\UnitTesting.tt” #>
<#@ import namespace=”System.Diagnostics” #>
<#+
‘ <copyright file=”CrudProcedureGeneratorTest.tt” company=”Microsoft”>
‘  Copyright © Microsoft. All Rights Reserved.
‘ </copyright> 

<TestClass> _
Public Class CrudProcedureGeneratorTest 

    <TestInitialize> _
    Public Sub TestInitialize() 

    End Sub 

    <TestCleanup> _
    Public Sub TestCleanup() 

    End Sub 

    <TestMethod> _
    Public Sub Test_Success()
        Assert.Inconclusive()
    End Sub 

    <TestMethod> _
    <ExpectedException(GetType(Exception))> _
    Public Sub Test_Exception()
        Assert.Inconclusive()
    End Sub 

End Class
#>

The first important thing to note is that this is a T4 file, which will be transformed by the code generation engine just like any other .tt file. This file uses an include directive to reference T4Toolbox\UnitTesting.tt, which contains the unit test runner itself, as well as assembly and import directives to reference the Visual Studio unit testing framework. The test runner will automatically run all unit tests when the code generation engine transforms this file.

Implement a test method

  • Modify CrudProcedureGeneratorTest.tt to look as shown below.
C#
<#@ template language=”C#v3.5hostspecific=”Truedebug=”True” #>
<#@ output extension=”log” #>
<#@ include file=”T4Toolbox.tt” #>
<#@ include file=”T4Toolbox\UnitTesting.tt” #>
<#@ import namespace=”System.Diagnostics” #>
<#@ include file=”CrudProcedureGenerator.tt” #>
<#+
[TestClass]
public class CrudProcedureGeneratorTest
{
    [TestMethod]
    public void Test_Validate_RequiresDatabaseName()
    {
        CrudProcedureGenerator generator = new CrudProcedureGenerator();
        generator.Run();
        TransformationAssert.HasError(generator, “DatabaseName”);
    }
}
#>

Visual Basic
<#@ template language=”VBv3.5hostspecific=”Truedebug=”True” #>
<#@ output extension=”log” #>
<#@ include file=”T4Toolbox.tt” #>
<#@ include file=”T4Toolbox\UnitTesting.tt” #>
<#@ import namespace=”System.Diagnostics” #>
<#@ include file=”CrudProcedureGenerator.tt” #>
<#+
<TestClass> _
Public Class CrudProcedureGeneratorTest
    <TestMethod> _
    Public Sub Test_Validate_RequiresDatabaseName()
        Dim generator As CrudProcedureGenerator = new CrudProcedureGenerator()
        generator.Run()
        TransformationAssert.HasError(generator, “DatabaseName”)
    End Sub
End Class
#>

Note that this example uses include directive to reference definition of the generator under test – CrudProcedureGenerator.tt.

  • Save CrudProcedureGeneratorTest.tt in Visual Studio editor to trigger the transformation process.

During transformation, the code generation engine will compile the code of CrudProcedureGeneratorTest in the same temporary assembly with CrudProcedureGenerator. When the compiled code runs, the code in T4Toolbox\UnitTesting.tt will start the test runner. The test runner will use reflection to locate all classes marked with TestClass attribute – CrudProcedureGeneratorTest in our example. It will then locate all methods of the test class marked with TestMethod attribute – such as Test_Validate_RequiresDatabaseName. For each test method, the test runner will create a new instance of the test class and call the test method. Finally, the test runner records detailed information about each test in the output .log file.

You can use all assert classes from the Visual Studio unit testing framework: Assert, CollectionAssert and StringAssert. T4 Toolbox also defines TransformationAssert class, which provides methods for assertions frequently used in T4 unit tests. For example, HasError method will check the Errors collection of the specified Generator or Template for a particular error and throw AssertFailedException if it cannot be found. Similarly, NoErrors will check the Errors collection to make sure it is empty.

  • In the Solution Explorer, open the CrudProcedureGeneratorTest.log which was generated by the unit test. You will see something similar to this:
Test_Validate_RequiresDatabaseName - Passed.

Troubleshooting test failures

Your unit tests will be likely to fail at some point, especially if you are using test-driven development approach while developing your code generators.

  • Change the following assertion in the Test_Validate_RequiresDatabaseName method to simulate an error in the unit test.
        TransformationAssert.HasError(generator, “DatabaseNameMisspelled”)

  • Save the CrudProcedureGeneratorTest.tt to trigger test execution. You will notice that the test runner reports unit test failures in the Error List window of Visual Studio.

Test failures are reported in Error List window

If you double-click the error, Visual Studio will automatically open the unit test file and place text caret in the line where the error occurred.

  • Open the CrudProcedureGeneratorTest.log file generated by the unit test.
Test_Validate_RequiresDatabaseName - Failed. Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException: Error 'DatabaseNameMisspelled' was expected during transformation
   at T4Toolbox.TransformationAssert.HasError(CompilerErrorCollection errors, String errorText)
   at T4Toolbox.TransformationAssert.HasError(Generator generator, String errorText)
   at Microsoft.VisualStudio.TextTemplating586E0B50DE26B04814EFC07CBF168F19.GeneratedTextTransformation.CrudProcedureGeneratorTest.Test_Validate_RequiresDatabaseName() in c:\Users\osych\Documents\Visual Studio 2008\Projects\T4Tutorial\T4TutorialCS\CrudProcedureGeneratorTest.tt:line 16

Notice that the output file contains detailed exception stack trace that can help you to identify more difficult problems. If this information is insufficient, you can debug the failed unit test by placing a breakpoint and stepping through its code, as described here.

Conclusion

Unit testing is a powerful technique for improving quality of code generators. It is especially important when code generators are created for use by other people. Automated testing of T4 code generators is challenging because code is compiled in a temporary assembly. The T4 Toolbox solves this challenge by providing a test runner which is compiled together with the code generator under test and its unit tests. This test runner supports a subset of the unit testing framework provided by Visual Studio, including the following attributes: TestClassAttribute, TestMethodAttribute, ExpectedExceptionAttribute, TestInitializeAttribute and TestCleanupAttribute. All assert classes are supported, including Assert, CollectionAssert and StringAssert.

In the next article in this series, we will discuss how to make code generators extensible and allow users to override code generation logic without having to make changes and maintain third-party code.

Download


Write a Comment

Take a moment to comment and tell us what you think. Some basic HTML is allowed for formatting.

Reader Comments

Updated source code to be compatible with version 9.10 of the T4 Toolbox.

Hi Oleg,

I’m using the T4 Toolbox (9.5.20.1) provided as part of Sharp Architecture 1.0.

With this version, I cannot use the TransformationAssert.HasError(genInstance, “PropertyName”). I believe it is expecting a template instance instead of a generator instance.

Is there any documentation anywhere on the Assert methods and the parameters?

Thanks.

Hi Oleg. Thanks for the great tutorial. I encountered a problem with this example. Both C# and VB version doesn’t generate any log after I save the unit test or try to “Run custom tool” on
Windows 7 x64
VS 2010 SP1
t4 editor 1.0.0
t4 toolbox 10.3.7.1.
Can you help, please?