This article is based on Beta 2 of Visual Studio 2010 Ultimate Edition. To follow the code example you may also want to install T4 Editor which provides color syntax highlighting and IntelliSense for text templates in Visual Studio.

UML Modeling in Visual Studio

Visual Studio 2010 Ultimate Edition includes a fully-featured UML (Unified Modeling Language) support, integrated into the IDE. You can create Use Case diagrams to summarize who uses your application or system, and what they can do with it, Class diagrams to describe logical aspects of data types and their relationships, Sequence diagrams to display interaction between classes, components and actors, Activity diagrams to document flow of an algorithm or a business process and Component diagrams to show structure of a system, and how it can be deployed to a logical environment.

In order to start using UML diagrams in Visual Studio 2010, you first need to create a Modeling Project.

Once you have the project created, you can add new diagrams to it.

Here is an example of class diagram you can create.

Model elements can be added to the diagram from Toolbox and configured using Properties window.

You may have noticed that these diagrams are very similar to the sample diagrams that were included with the DSL (Domain-Specific Language) Toolkit in Visual Studio 2008. While little has changed on the surface, DSL diagrams had a major limitation before Visual Studio 2010. Model information was stored with each diagram, limiting you to a single diagram per model. Needless to say, this limitation made it impossible to use DSL diagrams for all but simplest models.

In Visual Studio 2010, definitions of model elements are stored in a centralized repository as part of your Modeling Project; they can be displayed on multiple diagrams. You can browse contents of the repository using UML Model Explorer and add existing elements to new diagrams by dragging them from this window onto the design surface.

UML Profiles

While UML works well for conceptual analysis, the standard models are often too abstract to be useful for logical and physical design. UML Profiles can be used to extend the standard models and adapt them for a particular purpose.

In Visual Studio 2010, a UML Profile is a collection of stereotypes that can be applied to individual model elements. Each stereotype defines a set of attributes that extend definition of the model element it is applied to. For example, Visual Studio includes C# Profile that can be used to extend UML classes with attributes specific to C#.

In order to use stereotypes from a particular profile, first you need to select a package in the UML Model Explorer and apply the profile in the Properties window.

Once a profile has been specified for the package, you can apply the stereotypes defined by the profile to the elements within the package. Example below shows the C# Class stereotype applied to a UML class, extending it with C#-specific attributes, such as Is Partial and Is Static.

Custom UML Profiles

You may want to consider creating a custom UML profile if you rely heavily on modeling in your software development project. Database modeling in particular has been widely adopted by the IT industry, with most of the new projects using database models in some shape or fashion. Unfortunately, no standard UML profile is available for database modeling at the time of this writing. However, the process of creating such a profile in Visual Studio 2010 is relatively straightforward.

Visual Studio 2010 UML profiles are defined in XML files with .profile extension.

The profile definition consists of three sections: metaclasses, propertyTypes and stereotypes.

The metaclasses section lists all UML meta-classes that are extended by stereotypes in the profile. In practical terms, meta-class is a type of model element, such as Package, Class or Association. Meta-classes are identified using fully-qualified type names of the interfaces used in Visual Studio implementation. For example, IAssociation interface represents UML meta-class Association.

The propertyTypes section lists all types used by properties in stereotypes defined by the profile. Standard types, such as System.String can be specified using externalType element. You can also use enumerationType elements to define enumerated types, such as ForeignKeyRule.

<enumerationType name=“ForeignKeyRule“>
  <enumerationLiterals>
    <enumerationLiteral name=“NoAction” displayName=“No Action“/>
    <enumerationLiteral name=“Cascade” displayName=“Cascade“/>
    <enumerationLiteral name=“SetNull” displayName=“Set Null“/>
    <enumerationLiteral name=“SetDefault” displayName=“Set Default“/>
  </enumerationLiterals>
</enumerationType>

Finally, the stereotypes section contains the actual stereotype definitions. Each stereotype contains a metaclasses element, which specifies the UML meta-class this stereotype extends, and a properties element, which lists the properties that will be added to the target model element when the stereotype is applied to it. 

<stereotype name=“ForeignKey” displayName=“Foreign Key“>
  <metaclasses>
    <metaclassMoniker name=“/Sql2008DatabaseProfile/Microsoft.VisualStudio.Uml.Classes.IAssociation” />
  </metaclasses>
  <properties>
    <property name=“DeleteRule” displayName=“Delete Rule” defaultValue=“NoAction“>
      <propertyType>
        <enumerationTypeMoniker name=“/Sql2008DatabaseProfile/ForeignKeyRule“/>
      </propertyType>
    </property>
    <property name=“UpdateRule” displayName=“Update Rule” defaultValue=“NoAction“>
      <propertyType>
        <enumerationTypeMoniker name=“/Sql2008DatabaseProfile/ForeignKeyRule“/>
      </propertyType>
    </property>
  </properties>
</stereotype>

As you can see, the ForeignKey stereotype shown above can be applied to Associations in a UML model. This stereotype defines two properties – DeleteRule and UpdateRule, both of which use the ForeignKeyRule enumerated type defined in the propertyTypes section of our profile.

Notice that various elements in the profile definition have attributes called name and displayName. As you might expect, name specifies internal identifier you can use to access values programmatically, and displayName specifies human-readable text displayed by Visual Studio to the developer.

Sample Database Profile

A sample UML profile for database modeling is available for download at the end of this article. It allows you to extend UML class diagrams with stereotypes for modeling database schemas, tables, columns, primary and foreign keys in SQL Server 2008. Here is what a database diagram developed with this profile might look like.

The sample database profile contains the following stereotypes.

Profile Installation

The intended way to install custom UML Profiles is with the help of Visual Studio Extensions, new to Visual Studio 2010. The process is well documented on MSDN, simple and easy to follow. Unfortunately, as of Visual Studio 2010 Beta 2, VSIX deployment of custom profiles appears to be broken. In order to install the sample database profile, I had to resort to manually copying the .profile file to the following directory on a 64-bit Windows 7 installation:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Microsoft\Team Architecture\UmlProfiles

and modifying extension.vsixmanifest file located in this directory to look like this.

<?xml version=“1.0” encoding=“utf-8“?>
<Vsix Version=“1.0.0” xmlns=“http://schemas.microsoft.com/developer/vsx-schema/2010“>
  <Identifier Id=“Microsoft.VisualStudio.TeamArchitect.ProfileExtension“>
    <Name>Team Architecture Profile Extension</Name>
    <Author>Microsoft</Author>
    <Version>10.0</Version>
    <Description>Microsoft Visual Studio Team Architecture Profile Extension</Description>
    <Locale>1033</Locale>
    <SupportedProducts>
      <VisualStudio Version=“10.0“>
        <Edition>VSTS</Edition>
      </VisualStudio>
    </SupportedProducts>
    <SupportedFrameworkRuntimeEdition MinVersion=“4.0” MaxVersion=“4.0” />
    <SystemComponent>true</SystemComponent>
  </Identifier>
  <References />
  <Content>
    <CustomExtension Type=“Microsoft.VisualStudio.UmlProfile“>CSharp.Profile</CustomExtension>
    <CustomExtension Type=“Microsoft.VisualStudio.UmlProfile“>StandardProfileL2.Profile</CustomExtension>
    <CustomExtension Type=“Microsoft.VisualStudio.UmlProfile“>StandardProfileL3.Profile</CustomExtension>
    <CustomExtension Type=“Microsoft.VisualStudio.UmlProfile“>Sql2008Database.profile</CustomExtension>
  </Content>
</Vsix>

Notice the CustomExtension added in the end of the file, specifying Sql2008Database.profile as an extension of Microsoft.VisualStudio.UmlProfile type.

Code Generation

The simplest way to generate code from a UML model is with the help of T4 text templates. Text template is a simple code generator that can be added to any C# and Visual Basic project.

Here is the code you need to have in your template to load metadata from a UML model.

<#@ template debug=“true“ hostspecific=“true“ language=“C#“ #>
<#@ output extension=“.sql“ #>
<#@ assembly name=“Microsoft.VisualStudio.Uml.Interfaces.dll“ #>
<#@ assembly name=“Microsoft.VisualStudio.Uml.Extensions.dll“ #>
<#@ import namespace=“Microsoft.VisualStudio.Uml.Classes“ #>
<#@ import namespace=“Microsoft.VisualStudio.Uml.Extensions“ #>
<#
    string projectPath = Host.ResolvePath(@”..\ModelingProject\ModelingProject.modelproj”);
    IModelingProject project = ModelingProject.Load(projectPath);
#>

Notice the use of assembly and import directives to reference the Visual Studio assemblies and namespaces that provide API for accessing information in UML models. ResolvePath method of the Host object is used to determine absolute path to the modeling project and Load method of the ModelingProject class is used to open the UML repository.

Here is a complete template that generates SQL scripts for creating tables based on the UML class model extended by our database profile.

<#@ template debug=“true“ hostspecific=“true“ language=“C#“ #>
<#@ output extension=“.sql“ #>
<#@ assembly name=“System.Core“ #>
<#@ assembly name=“Microsoft.VisualStudio.Uml.Interfaces.dll“ #>
<#@ assembly name=“Microsoft.VisualStudio.Uml.Extensions.dll“ #>
<#@ import namespace=“System.Linq“ #>
<#@ import namespace=“Microsoft.VisualStudio.Uml.Classes“ #>
<#@ import namespace=“Microsoft.VisualStudio.Uml.Extensions“ #>
<#
    string projectPath = this.Host.ResolvePath(@”..\ModelingProject\ModelingProject.modelproj”);
    IModelingProject project = ModelingProject.Load(projectPath);
    foreach (IType t in project.Store.Root.OwnedTypes)
    {
        IClass c = t as IClass;
        if (c == null) continue;
#>
create table <#= GetSchema(project.Store.Root) #>.<#= c.Name #>
(
<#
        this.PushIndent(“\t”);
        foreach (IProperty p in c.OwnedAttributes)
        {
            IStereotypeInstance column = p.AppliedStereotypes.Where(s => s.Name == “Column”).First();
            this.WriteLine(p.Name + ” “ + GetDataType(column) + GetNull(column) + GetIdentity(column) + “,”);
        }
        this.PopIndent();
#>
);
<#
    }
#>
<#+
    private string GetSchema(IPackage package)
    {
        IStereotypeInstance schema = package.AppliedStereotypes.Where(s => s.Name == “Schema”).First();
        IStereotypePropertyInstance schemaName = schema.PropertyInstances.Where(p => p.Name == “Name”).First();
        return schemaName.Value;
    } 

    private string GetDataType(IStereotypeInstance column)
    {
        IStereotypePropertyInstance dataType = column.PropertyInstances.Where(p => p.Name == “DataType”).First();
        return dataType.Value;
    } 

    private string GetNull(IStereotypeInstance column)
    {
        IStereotypePropertyInstance allowNulls = column.PropertyInstances.Where(p => p.Name == “AllowNulls”).First();
        return bool.Parse(allowNulls.Value) == true ? string.Empty : ” not null”;
    } 

    private string GetIdentity(IStereotypeInstance column)
    {
        IStereotypePropertyInstance isIdentity = column.PropertyInstances.Where(p => p.Name == “IsIdentity”).First();
        return bool.Parse(isIdentity.Value) == true ? ” identity” : string.Empty;
    }
#>

This code uses a foreach loop to iterate through the list of types owned by the Root model in our project and generates a create table script for each class. Notice the helper methods, such as GetSchema and GetDataType. They use AppliedStereotypes property of the model element to find the stereotype defined in our database profile, such as Schema or Column. Having found the stereotype instance, its properties properties are accessed through the PropertyInstances collection.

Here is the code generated by this text template from our UML model.

create table dbo.Customer
(
    FirstName nvarchar,
    LastName nvarchar,
    StreetAddress nvarchar,
    City nvarchar,
    State nvarchar,
    PostalCode nvarchar,
    Id int not null identity,
);
create table dbo.Product
(
    Name nvarchar,
    Price nvarchar,
    Id int not null identity,
);
create table dbo.Order
(
    Date nvarchar,
    TotalAmount nvarchar,
    Id int not null identity,
    CustomerId int,
);
create table dbo.OrderItem
(
    Id int not null identity,
    OrderId int,
    ProductId int,
);

UML vs. DSL

UML models are high-level abstractions, typically representing systems at conceptual or logical levels. This is an intentional, desired aspect, as it allows you to model a wide variety of of systems without tying the design to any particular platform or implementation. While high-level models serve as an excellent communication tool during envisioning and design phases, during implementation high-level models quickly become outdated, causing confusion. While automated code generation and reverse engineering of models is possible, the process is complex, fragile and requires additional effort.

Comprehensive support for UML is a significant change in direction of Microsoft’s approach to modeling. You may remember that when Domain-Specific Language tools were announced a few years ago, the emphasis was on practical application of modeling throughout the software development lifecycle. It was suggested that models should be considered equal to other source code artifacts, such as .cs/.vb files and .aspx pages. Typical DSL models are lower-level abstractions, representing a system at logical or physical levels. They are predominantly used for code generation and rely on partial classes and inheritance to avoid reverse engineering model information from code back to the model. Examples of DSL models in Visual Studio include LINQ to SQL and Entity Framework designers.

While these approaches to modeling appear to be different, in reality they complement each other, providing us with a full range of modeling capabilities from conceptual and logical, down to physical level. UML models are useful during early stages of a software development project and DSL models work best during implementation and maintenance of the system. It is interesting to note that UML models are actually implemented using DSL tools, so from Visual Studio prospective, UML is simply a DSL for the “unified modeling” domain.

Conclusion

As you can see, Visual Studio 2010 Ultimate Edition provides powerful UML modeling tools that can be used for conceptual and logical analysis during software development. UML Profiles can be used to extend standard models and allow modeling information systems at physical level. A limited number of UML profiles is expected to ship with Visual Studio 2010. Custom profiles can be developed to tailor UML models to the needs of individual projects. Combined with T4 Text Templates, this gives developers a powerful tools for model-driven development of information systems, where large portions of the system can be modeled in UML and generated with T4. 

References

• MSDN, Developing Models for Software Design
• Cameron Skinner, UML Profiles and Visual Studio 2010 Ultimate: Part One
• Istvan Novak, Deploying Visual Studio 2010 Extensions, Part 1
• Tim Fischer, How To: Generate Code from Team System UML Diagrams in VS 2010 Beta 2 
• Oleg Sych, Text Template Transformation Toolkit

Download

Source code

©2010 Oleg Sych. All Rights Reserved.

Design of Entity Framework has been inspired by the traditional methodology of relational database design which suggests creating a conceptual (class) model, transforming it into logical (relational) model and finally into physical database model. While these activities are typically performed during design and initial implementation of the information system, the Entity Framework implements the physical model, makes the conceptual model a first-class citizen and explicitly defines the mapping model.

• Conceptual model defines entities and their relationships from the application point of view. This model is defined using Conceptual Schema Definition Language (CSDL) and stored in a separate .csdl file or as part of a combined .edmx file in your project.
• Storage model defines entities and their relationships from the database point of view. This model is defined using Store Schema Definition Language (SSDL) and stored in a separate .ssdl file or as part of a combined .edmx file in your project.
• Mapping model defines how conceptual model translates into storage model. It is defined using Mapping Specification Language (MSL) and stored in a separate .msl file or as part of a combined .edmx file in your project.

The Entity Framework uses these XML-based models to transform create, read, update, and delete operations against entities and relationships in the conceptual model to equivalent operations in the data source. The storage and mapping models can change as needed without requiring changes to the conceptual model, data classes, or application code. Because storage models are provider-specific, you can work with a consistent conceptual model across various data sources. The Entity Data Model even supports mapping entities in the conceptual model to stored procedures in the data source.

What is wrong with this picture?

Entity Framework was designed to overcome the object-relational impedance mismatch and allow you to hide a difficult-to-use legacy database schema behind a modern, object-oriented entity model and make developers more productive. This description almost sounds to good to be true. Unfortunately, just like other frameworks designed to solve complex problems, the Entity Framework tends to make simple things difficult.

Having three separate database layers (or models) in the Entity Data Model means having each entity represented three times, each from a different point of view. Every time you need to make a change in your database model, which also needs to be implemented in the application, you have to make it three times. Take adding a new column to a table as an example. No matter in which order you do it, you still have to add a new property to the conceptual model, a new column to the storage model and a new mapping from one to the other. This task is tedious and error prone even with the best modeling tools imaginable. Unfortunately, Entity Framework designer limits you to a single diagram, which gradually becomes more difficult to use as the number of entities increases, becoming virtually useless when it exceeds 50.

In simplistic terms, this all means that Entity Data Model is three times as complex as a traditional database model. To implement all but trivial logic, either one developer has to understand all three models or you have to have two or more people involved in implementing it. In the end, you have to spend three times more effort to develop and maintain the EDM, you are three times more likely to make mistakes, i.e. the cost of having the EDM in your solutions is three times higher than the cost of a solution that uses a single hypothetical data model.

What is the alternative?

Majority of information systems in existence today are built using relational databases and data-oriented code. A tremendous amount of tooling, knowledge and experience has been developed over the past two decades by the industry, making low-cost development resources readily available. Although solving the object-relational impedance mismatch sounds very good in theory, in practice, the overall cost of such a solution is typically higher, in part due to additional effort required to maintain three separate database models and in part due to additional knowledge and experience required from developers.

However, the existing as well as new information systems could benefit tremendously from having a modern, strongly-typed, LINQ-enabled, automatically generated data access layer (DAL). Such a DAL would allow to significantly reduce the amount of plumbing code developers have to write today with DbConnection, DbDataReader and DataSet classes without requiring significant changes to the application architecture. In other words, as long as the effort required to maintain such a DAL is less than the effort required to maintain the data access code manually, we can take advantage of the language and data access enhancements offered by the current version of the .NET framework while reducing the overall cost of the information systems.

Entity Framework DAL generator in T4 Toolbox

In order to implement an automatically generated DAL based on Entity Framework today, you would have to use EdmGen. Unfortunately, it does not give you access to some of the options supported by the framework. The new version of T4 Toolbox includes a code generator that encapsulates the functionality built into Entity Framework and makes it available in T4-based code generation scripts. This code generator is based on the idea of using database modeling tools built into Visual Studio, which are easier to use and unlike Entity Framework’s own designer, are proven to work on the number of tables typically found in real-world databases. In the rest of this article we will review the process of using this code generator to illustrate its pros and cons.

Getting Started

In order to follow this sample, you need to have Visual Studio 2008 or 2010, Standard Edition or higher, SQL Server Express 2005 or later, T4 Toolbox and T4 Editor installed on your computer.

• Create a new C# or Visual Basic console application project.
• Add a new item to your project and select the Code Generation->Entity Framework Database template in the Add New Item dialog.

• Enter the desired database name and click Add.

This template adds a new, empty SQL Express database to your project. Unfortunately, Visual Studio automatically displays the the Data Source Configuration Wizard whenever you add a .mdf file to a project. Normally, you would use it to create a data access layer with built-in visual designers. In this case, we have the entire data access layer generated automatically for us.

• Click Cancel in the Data Source Configuration Wizard.

Notice that a number of files have been added to your project, including a .mdf file (SQL database), a .ldf file (SQL log), a .tt file (code generator), a .cs/.vb file (entity classes and object context), a .Views.cs/.vb file (precompiled entity views), a .csdl file (conceptual model), a .ssdl file (storage model) and a .msl file (mapping model). If you are using Visual Basic, you will need to click "Show All Files” button in solution explorer in order to see the files nested under .mdf and .tt.

Database Modeling

Visual Studio includes a subset of database modeling functionality from SQL Server Management Studio, including database diagrams, table designers, key and relationship designers.

• Double-click the .mdf file in Solution Explorer (shown above). This will cause the database to be opened in Server Explorer (shown on the left).
• Right-click Database Diagrams in Server Explorer and select Add New Diagram from the context menu.
• The first time you add a diagram to a database, Visual Studio will prompts you to create database objects required for diagramming. Click Yes when prompted.

Use the diagram to create a new database model.

Database diagrams allow you to quickly model tables and their relationships. You can use the Properties Window of Visual Studio to configure individual tables and columns selected on the design surface. The toolbar and context menu provide access to commands that allow you to model keys, relationships, indexes, unique and check constraints, as well as customize the look of your diagram for display and print, add text annotations and more. Also, unlike with Entity Framework designer, you can create additional diagrams when the number of tables is too big to fit on a single diagram. This seemingly trivial feature is crucial for large database models.

Having the model database included in your project has several important benefits. It is placed under source control with the rest of the files in your project, giving you the ability to go back to previous versions when necessary. Although it may seem excessive to check in an entire database in source control, the size of a .mdf file is comparable to the size of a similar database model stored in a Visio .vsd file or an Enterprise Architect .eap file. You can also use the database file itself to store any seed data you need and with “Copy to Output Directory” option turned on, the database deployment becomes a simple xcopy on greenfield projects.

• Make sure to save changes you make in any database diagram or designers you open before you proceed to the next step of generating code.

Code Generation

• Double-click the .tt file in Solution Explorer (shown above) to open it in the T4 Editor.

C#

<#@ template language="C#" hostspecific="True" #>
<#@ output extension="cs" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="T4Toolbox\EntityFramework.tt" #>
<#
    var generator = new EntityFrameworkGenerator();
    generator.DatabaseFile = "Database1.mdf";
    generator.Run();
#>

VB

<#@ template language="VB" hostspecific="True" #>
<#@ output extension="vb" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="T4Toolbox\EntityFramework.tt" #>
<#
    Dim generator As New EntityFrameworkGenerator()
    generator.DatabaseFile = "Database1.mdf"
    generator.LanguageOption = LanguageOption.GenerateVBCode
    generator.Run()
#>

This is a T4 code generation script that creates a new instance of EntityFrameworkGenerator, configures its properties and uses it to generate all model and code files required for a ready-to-use data access layer based on Entity Framework. Note that this script is not directly associated with the database file in the Visual Studio project and has to be run manually.

• Right-click the .tt file in Solution Explorer and select Run Custom Tool from its context menu to trigger the code generation.
• Double-click the  generated .cs/.vb file in Solution Explorer to open it in the Visual Studio editor. If you are using Visual Basic, you will need to click “Show All Files” in the toolbar of Solution Explorer in order to see this file.

C#

VB

This generated file contains definitions of both entity classes and object context required to access the database using LINQ to Entities.

Also note the other generated code file, in our example Database1.Views.cs/.vb. This file contains pre-generated views, which significantly improve reduce the amount of time required to connect to a database using an Entity Framework object context for the first time. This can take tens of seconds even on a medium-size database with a hundred of tables. While this performance hit does not have a meaningful effect on applications in production due to Entity Framework caching the views on per-AppDomain basis, this delay is a significant problem for developers who need to recompile and restart the application frequently. The Entity Framework designer does not generate views automatically, causing much frustration for developers and, perhaps, being the main reason for Entity Frameworks reputation of having poor performance.

Coding

At this point, you have everything you need to start coding against your new database using LINQ to Entities.

• Double-click Program.cs or Module.vb in Solution Explorer.
• Enter the following code in the editor.

C#

using System;
using System.Linq; 

namespace EfDalSampleCS
{
  class Program
  {
    static void Main(string[] args)
    {
      var context = new Database1Entities();
      context.AddToProducts(new Product() { ProductName = "Internet Explorer", UnitPrice = 0 });
      context.AddToProducts(new Product() { ProductName = "Windows 7 Ultimate", UnitPrice = 319.99M });
      context.SaveChanges(); 

      var products = from p in context.Products select p;
      foreach (var product in products)
      {
          Console.WriteLine("{0} ({1})", product.ProductName, product.UnitPrice);
      }
    }
  }
}  

VB

Imports System
Imports System.Linq
Imports EfDalSampleVB.EfDalSampleVB 

Module Module1
    Sub Main()
        Dim context As New Database1Entities
        context.AddToProducts(New Product() With {.ProductName = "Internet Explorer", .UnitPrice = 0})
        context.AddToProducts(New Product() With {.ProductName = "Windows 7 Ultimate", .UnitPrice = 319.99})
        context.SaveChanges() 

        Dim products = From p In context.Products Select p
        For Each product In products
            Console.WriteLine("{0} ({1})", product.ProductName, product.UnitPrice)
        Next
    End Sub
End Module 

 

• Press F5 to run the program. You may want to set a breakpoint and observe execution of the program in Visual Studio debugger.

A couple of things are worth explaining. How does this code know which database to connect to? The answer lies in the default constructor of Database1Entities class, which is generated to use a connection string with the same name. This connection string was automatically added to the configuration file when we added Database1.mdf to our project.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <connectionStrings>
    <add name="Database1Entities"
      connectionString=
      "
        metadata=res://*/Database1.csdl
                |res://*/Database1.ssdl
                |res://*/Database1.msl;
        provider=System.Data.SqlClient;
        provider connection string=
        ‘
          Data Source=.\SQLEXPRESS;
          AttachDbFilename=|DataDirectory|\Database1.mdf;
          Integrated Security=True;
          User Instance=True
        ‘
      "
      providerName="System.Data.EntityClient" />
  </connectionStrings>
</configuration>

Metadata section of the connection string specifies that .csdl, .ssdl and .msl files are embedded as resources in the executable. Provider section of the connection string instructs Entity Framework to use ADO.NET SQL Client to connect to the database. The provider connection string section contains the traditional connection string and indicates that SQL Server Express edition will be used to automatically attach the database file located in DataDirectory. For windows applications (such as the console app we have here), DataDirectory token refers to the directory where the  executable is located. Our database file is copied to the output directory because the “Copy to Output Directory” option was set to “Copy Always” for the .mdf file in our project.

Customizing the list of database objects exposed by DAL

You may have noticed that among the generated entity classes, we also have sysdiagrams, which was actually the table created by SQL Server when we added the first diagram to our database. Unless you are planning to access this table from your application, it would be best to remove it from the data access layer. You can can do that by adding a filter to the code generator.

• Add the following code to the .tt file before generator.Run() method is called.

C#

  generator.Filters.Add(
    new EntityStoreSchemaFilterEntry(
      null,   // catalog
      null,   // schema
      "sys%", // name
      EntityStoreSchemaFilterObjectTypes.Table,
      EntityStoreSchemaFilterEffect.Exclude));

VB

  generator.Filters.Add( _
    New EntityStoreSchemaFilterEntry( _
      Nothing, _
      Nothing, _
      "sys%",  _
      EntityStoreSchemaFilterObjectTypes.Table, _
      EntityStoreSchemaFilterEffect.Exclude))

This code creates a filter entry object that indicates that all tables whose name starts with “sys” should be excluded from the generated DAL. EntityFrameworkGenerator class contains a property called Filters to which you can add one or more filter entries and make sure that only appropriate database objects are accessible through the DAL. For complete details, please refer to EntityStoreSchemaFilterEntry, EntityStoreSchemaFilterObjectTypes and EntityStoreSchemaFilterEffect on MSDN.

Unfortunately, not all of the possible combinations of object types and filter effects are supported by the underlying Entity Framework generator. For example, creating a filter entry to include functions in the DAL has no effect because the functions are only automatically generated in the storage model (.ssdl) and not in the conceptual model (.csdl). In other words, you cannot access stored procedures using the DAL produced by this code generator in its current version.

Customizing pluralization of entity class names and set names

Entity Framework 4.0 in Visual Studio 2010 provides an option to automatically convert table names to singular form in names of entity classes and to plural form in names of entity set properties. Without this automatic conversion, both entity name and set name would be either in plural form or singular form, depending on the naming convention you use in your database model. This poses a significant limitation in Entity Framework 3.5 in Visual Studio 2008, making the resulting DAL more confusing. A complete discussion of pluralization by the Entity Framework team is available on their blog.

The code generator in T4 Toolbox has the pluralization turned on by default in Visual Studio 2010. If you need to generate the DAL without pluralization, perhaps for compatibility reasons, you can turn this option off by adding the following line to the .tt file before generator.Run() method is called.

C#

  generator.Pluralize = false;

VB

  generator.Pluralize = False

This option is not available in Visual Studio 2008 because the underlying code generator in Entity Framework does not support it.

Customizing generation of foreign key properties

Entity Framework 4.0 in Visual Studio 2010 also provides an option to generate and use foreign key properties (such as Order.ProductId) in addition to navigation properties (such as Order.Product). Detailed discussion of foreign key properties is also available on the Entity Framework team blog.

The code generator in T4 Toolbox offers an option to generate foreign key properties, however you need to turn it on explicitly, by adding the following line to the .tt file before generator.Run() method is called.

C#

  generator.GenerateForeignKeyProperties = true;

VB

  generator.GenerateForeignKeyProperties = True

This option is not available in Visual Studio 2008 because the underlying code generator in Entity Framework does not support it.

Customizing location of the model database

When storing the model database under source control is not desirable on your project, you can change the code generator to use a standalone database as the model.

• Change the .tt file to specify a connection string instead of the database file.

C#

<#@ template language="C#" hostspecific="True" #>
<#@ output extension="cs" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="T4Toolbox\EntityFramework.tt" #>
<#
  EntityFrameworkGenerator generator = new EntityFrameworkGenerator();
  generator.ConnectionString = "Server=.;Database=Northwind;Integrated Security=True";
  generator.Run();
#>

VB

<#@ template language="VB" hostspecific="True" #>
<#@ output extension="vb" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="T4Toolbox\EntityFramework.tt" #>
<#
  Dim generator As New EntityFrameworkGenerator()
  generator.ConnectionString = "Server=.;Database=Northwind;Integrated Security=True"
  generator.LanguageOption = LanguageOption.GenerateVBCode
  generator.Run()
#>

When you save the modified .tt file, the DAL will be regenerated based on the metadata in the database you specified in the connection string. By default, database name determines the name of the generated object context class. Changing the name of the database from Database1 to Northwind will change the name of the context class from Database1Entities to NorthwindEntities.

• Rename the .tt file to match the name of the database (Northwind.tt in our example).

Although this is not required, renaming the code generation script will make the names of the generated .csdl, .ssdl and .msl files match the name of the database and make the solution easier to understand.

• Update the configuration file to match the changes made in generated code.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <connectionStrings>
    <add name="NorthwindEntities"
      connectionString=
      "
        metadata=res://*/Northwind.csdl
                |res://*/Northwind.ssdl
                |res://*/Northwind.msl;
        provider=System.Data.SqlClient;
        provider connection string=
        ‘
          Server=.;
          Database=Northwind;
          Integrated Security=True;
        ‘
      "
      providerName="System.Data.EntityClient" />
  </connectionStrings>
</configuration>

In particular, the name of the connection string has to match the new name of the object context class (NorthwindEntities in this example); file names in the metadata section of the connection string have to match the names of the generated model files (Northwind.csdl, Northwind.ssdl and Northwind.msl in this example); and finally, the provider connection string section of the connection string has to point to the actual database in your current environment.

Note that database connection string is present in both .tt and .config files. In this example, the same Northwind database on local SQL server is used during both code generation and execution. You may want to use different databases for these purposes, perhaps an empty local database for code generation and a shared database for development and testing. In that case, make sure to use appropriate connection strings in the .tt and .config files.

Other customization options

Additional details about customization options are available in the T4 Toolbox documentation installed as part of the Visual Studio 2008 help collection. If you have it installed, you can click here to open the list of properties available for you to use in the code generation script.

Conclusion

ADO.NET Entity Framework allows you to access database in object-oriented manner at the cost of additional effort required to maintain conceptual, storage and mapping model. While sounding good in theory, solving the object-relational impedance mismatch in practice typically increases the cost of information systems due to additional knowledge, experience and effort it requires with tools available today. By implementing an automatically-generated, modern, strongly-typed, LINQ-enabled data access layer based on Entity Framework, we can improve developer productivity and reduce the system development cost without introducing the overhead and complexity of fully featured object-relational mapping. Unlike the tools built into Entity Framework, the database modeling functionality in Visual Studio and the code generator described in this article allow you to use Entity Framework effectively with more than just a few dozen tables. Unfortunately, due to limitations in the current version of Entity Framework, the solution described in this article does not allow accessing stored procedures and functions through the automatically generated DAL, making it inferior compared to a similar DAL based on LINQ to SQL today.

Download

Sample source code

©2010 Oleg Sych. All Rights Reserved.

You can watch the recording on Channel 9. In case you need to fast forward, here is the original outline I created for this session.

• Template Syntax and Transformation Process (5 minutes)
• Demo – Create simple template (15 minutes)
• Processing Directives (10 minutes)
• Demo – Template debugging (5 minutes)
• Template Deployment (5 minutes)
• Custom Tool Integration (5 minutes)
• Pre-compiled templates (5 minutes)
• What’s new in 2010 (5 minutes)

Unfortunately, the code example for this session was originally developed using Visual Studio 2010 Beta 1. In Beta 2, the APIs for accessing UML metadata are different and much easier to use. Here is a working template, inspired by Tim Fischer’s example:

<#@ template debug="true" hostspecific="true" language="C#" #>
<#@ output extension=".cs" #>
<#@ assembly name="Microsoft.VisualStudio.Uml.Interfaces.dll" #>
<#@ assembly name="Microsoft.VisualStudio.Uml.Extensions.dll" #>
<#@ import namespace="Microsoft.VisualStudio.Uml.Classes" #>
<#@ import namespace="Microsoft.VisualStudio.Uml.Extensions" #>
using System;

namespace ClassLibrary
{
<#
    string projectPath = this.Host.ResolvePath(@"..\ModelingProject\ModelingProject.modelproj");
    using (IModelingProject project = ModelingProject.Load(projectPath))
    {
        foreach (IType t in project.Store.Root.OwnedTypes)
        {
            IClass c = t as IClass;
            if (c == null) continue;
#>
    public class <#= c.Name #>
    {
<#
        this.PushIndent("\t\t");
        foreach (IProperty p in c.OwnedAttributes)
        {
            this.WriteLine("public " + p.Type.Name + " " + p.Name + ";");
        }
        this.PopIndent();
#>
    }
<#
        }
    }
#>
}

©2010 Oleg Sych. All Rights Reserved.

By default in T4, the code generation (a.k.a. template transformation) occurs when you save a template (.tt) file or select Run Custom Tool from its context menu in Solution Explorer. This works very well while you are actively developing the code generator, making changes in the template file and saving it to see the generated code. However, once development of the code generator itself is finished and your entire development team begins using it, this way of triggering code generation becomes less than ideal.

If your template generates code from metadata stored in a separate file, such as the LINQ to SQL .dbml file, it is easy for a developer to change the input file and forget to regenerate the code using the code generator in a separate T4 file. This mistake can lead to unnecessary frustration and wasted effort, especially when a T4-based code generator replaces a built-in code generator, such as MSLinqToSqlGenerator, which generates code automatically whenever the input file is saved.

You can eliminate the waste of effort and frustration by triggering your T4-based code generator automatically, whenever the input file is saved. This can be accomplished using a proven Visual Studio extensibility mechanism – Custom Tools. In this article, we will review how you can create your own custom tool or simply use the one provided by the T4 Toolbox.

Custom Tools

A custom tool - sometimes referred to as a single file generator - can be used to extend the Visual Basic, Visual C#, and Visual J# project systems in Visual Studio. A custom tool is a COM component that implements the IVsSingleFileGenerator interface. Using this interface, a custom tool transforms a single input file into a single output file. The result of the transformation may be source code, or any other output that is required. T4 itself provides a custom tool called TextTemplatingFileGenerator, which is automatically associated with .tt files and performs template transformation. Before you can use a custom tool, you must register it with the system or in the Visual Studio local registry. You can find complete details about custom tools and single file generators in Visual Studio SDK.

T4-Based Custom Tool

One way to automate T4-based code generation, is to create a custom tool that will be associated with your input file. When the input file is modified, Visual Studio will invoke this custom tool; the tool will load a T4 template either from an embedded resource or an external file and use ITextTemplating service to generate the code.

There are different ways to implement such custom tool. If you are building a modeling solution using Domain-Specific Language tools, Jean-Mark Prieur provides an excellent example in Part 4 of the DSL Tools Lab. Alternatively, you can follow instructions in Visual Studio SDK, which requires more work, but does not have a dependency on DSL toolkit. Either way, you will want to take advantage of the built-in functionality provided by T4 and inherit your custom tool from the TemplatedCodeGenerator as Jean-Mark suggests.

Although implementing a T4-based custom tool is not difficult, the custom tool requires registration, which means having an installation program. If you are a tool developer, this is not a problem – you probably already have one. If you are an application developer, this is just another bit of extra work that doesn’t directly contribute to your product, the extra work you want to avoid.

T4ScriptFileGenerator

T4 Toolbox (version 9.10 or later) provides a ready-to-use custom tool called T4ScriptFileGenerator. You can associate it with a particular file using Solution Explorer and Properties window as shown below.

First time you save the input file (Northwind.dbml in our example) after assigning T4ScriptFileGenerator as its Custom Tool, it will create an empty .tt file similar to this.

<#@ template language="C#" hostspecific="True" debug="True" #>
<#@ output extension="txt" #>
<#@ include file="T4Toolbox.tt" #>

Note that the new .tt file has the usual TextTemplatingFileGenerator custom tool associated with it.

When you add the actual code generation logic and save this file, T4 engine will transform it automatically, as you would expect. In this example, we will modify Northwind.tt file to look like this.

<#@ template language="C#v3.5" hostspecific="True" #>
<#@ output extension="log" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="T4Toolbox\LinqToSql.tt" #>
<#
    LinqToSqlGenerator generator = new LinqToSqlGenerator();
    generator.DbmlFile = "Northwind.dbml";
    generator.Run();
#>

Now if we go back and modify our input file, Northwind.dbml, the T4ScriptFileGenerator will reuse the existing T4 script, Northwind.tt, and trigger the code generation. In other words, the Northwind.tt file is being transformed automatically whenever we change the input file, Northwind.dbml.

This configuration process is easy to follow and once you are done, simply check your project into your source control and the custom tool configuration will be preserved in the project file itself. To begin using this code generator, other developers on your team simply need to have T4 Toolbox installed on their computers and get latest version of the project source code.

However, if you are building T4-based code generators that can be created and configured by others, perhaps your development team or external customers, you may want to save them the trouble of performing these steps manually every time they need to create a new code generator.

T4ScriptFileGeneratorWizard

T4 Toolbox (version 9.12 or later) also allows you to automatically associate T4ScriptFileGenerator with an input file using a Visual Studio Project Item Template. Simply put both the input file and the code generation script (.tt) file in the template as described in the article mentioned above and provide a VSTEMPLATE file similar to this.

<VSTemplate Version="2.0.0" xmlns="http://schemas.microsoft.com/developer/vstemplate/2005" Type="Item">
  <TemplateData>
    <DefaultName>DataClasses.dbml</DefaultName>
    <Name>LINQ to SQL Model</Name>
    <Description>LINQ to SQL entity classes and schema objects.</Description>
    <ProjectType>CSharp</ProjectType>
  </TemplateData>
  <TemplateContent>
    <References>
      <Reference><Assembly>System</Assembly></Reference>
      <Reference><Assembly>System.Core</Assembly></Reference>
      <Reference><Assembly>System.Data.Linq</Assembly></Reference>
    </References>
    <ProjectItem SubType="Designer" TargetFileName="$fileinputname$.dbml" ReplaceParameters="true">Dbml.xml</ProjectItem>
    <ProjectItem SubType="" TargetFileName="$fileinputname$.tt" ReplaceParameters="true">LinqToSqlModel.tt</ProjectItem>
  </TemplateContent>
  <WizardExtension>
    <Assembly>T4Toolbox, Version=9.11.21.1, Culture=neutral, PublicKeyToken=7e313accbcce84dc</Assembly>
    <FullClassName>T4Toolbox.VisualStudio.T4ScriptFileGeneratorWizard</FullClassName>
  </WizardExtension>
</VSTemplate>

Notice the use of <WizardExtension/> to associate the T4ScriptFileGenerationWizard with this project item template. This wizard extension will assign T4ScriptFileGenerator as a custom tool to the root item unfolded by the template - $fileinputname$.dbml or Northwind.dbml file in this example. If the project item template contains a T4 script file with the matching name ($fileinputname$.tt), the wizard extension will also make sure the code in it is preserved and used for code generation. If the project item template does not contain a T4 script file, an empty .tt file will be created instead.

Remember that T4 Toolbox has a separate assemblies for Visual Studio 2008 (T4Toolbox.dll) and Visual Studio 2010 (T4Toolbox.10.0.dll). The VSTEMPLATE example above is for Visual Studio 2008. In 2010, you have to change the <WizardExtension> to look like this.

<WizardExtension>
  <Assembly>T4Toolbox.10.0, Version=9.11.21.1, Culture=neutral, PublicKeyToken=7e313accbcce84dc</Assembly>
  <FullClassName>T4Toolbox.VisualStudio.T4ScriptFileGeneratorWizard</FullClassName>
</WizardExtension>

As you have already guessed, T4 Toolbox uses T4ScriptFileGenerator and T4ScriptFileGeneratorWizard in its LINQ to SQL model sample code generator for C#. You can see it in action yourself.

Conclusion

Although the built-in mechanism for triggering template transformation in T4 works well during development of a code generator, it often leads to mistakes during its normal use, when developers primarily modify the code generation model or input file. There are different ways of triggering the code generation automatically when the input file changes. In this article we discussed one of these approaches, based on Custom Tools. While it is possible to completely hide code generation from developers with this approach, doing so would prevent developers from being able to take advantage of rich template-based customization capabilities offered by T4. The T4ScriptFileGenerator in T4 Toolbox allows you get the best of both options: have the code generation occur automatically when the input file changes and still allow developers to customize the code generator by changing the T4 template.

©2010 Oleg Sych. All Rights Reserved.

The set of project item templates has been extended to support preprocessed templates and generators. Here is what you will see when you add a new item to a Visual Basic project.

This list contains familiar project item templates, such as Template, Generator and Unit Test. Script is the project item template that was previously named File. It is a traditional .tt file that is meant to include other, partial .tt files such as Template and Generator. Hopefully the new name Script makes more sense than File.

The list of project item templates for Visual C# projects is similar and also includes ready-to-use code generators written in C#. It is the same with the exception of the Enum SQL View, which at this time works only in Visual Studio 2008.

Preprocessed Templates and Generators

Preprocessed Template and Preprocessed Generator project item templates, which are similar to the traditional Template and Generator respectively, were added take advantage of the new capability offered by Visual Studio 2010 to preprocess .tt files and compile T4-based code generators in .NET assemblies.

When you add a Preprocessed Template to your project, two files get created - a preprocessed .tt file and a partial .cs or .vb file similar to the ones below.

Template.tt (C#)

<#@ template language="C#" inherits="T4Toolbox.Template" #>

Template.partial.cs

namespace ClassLibrary1
{
    using System;
    using T4Toolbox;

    public partial class Template1
    {
        protected override void Validate()
        {
            this.Warning("Template properties have not been validated");
        }
    }
}

Template.tt (Visual Basic)

<#@ template language="VB" inherits="T4Toolbox.Template" #>

Template.partial.vb

Imports T4Toolbox

Partial Public Class Template1
    Inherits Template

    Protected Overrides Sub Validate()
        Me.Warning("Template properties have not been validated")
    End Sub

End Class

For preprocessed .tt files, T4 doesn’t generate actual output; instead it generates the code generation template itself. Here is a cleaned-up version of what you will see if you open the files generated by TextTemplatingFilePreprocessor.

Template.cs

namespace ClassLibrary1
{
    using System;
    using System.CodeDom.Compiler;

    #line 1 "C:\…\Template1.tt"
    [GeneratedCodeAttribute("Microsoft.VisualStudio.TextTemplating", "10.0.0.0")]
    public partial class Template1 : T4Toolbox.Template
    {
        public override string TransformText()
        {
            return this.GenerationEnvironment.ToString();
        }
    }

    #line default
    #line hidden
}

Template.vb

Imports System
Imports System.CodeDom.Compiler

<GeneratedCodeAttribute("Microsoft.VisualStudio.TextTemplating", "10.0.0.0")> _
Partial Public Class Template1
    Inherits T4Toolbox.Template
    Public Overrides Function TransformText() As String
        Return Me.GenerationEnvironment.ToString
    End Function
End Class

Together, Template and Template.partial files define a single class that descends from T4Toolbox.Template which I assume is already familiar to the reader. In a traditional T4 Toolbox Template, you would place all logic in a single .tt file, using a mix of text blocks and class feature blocks. With a preprocessed template, you will typically want to use the .tt file only for the logic that requires text templating. Non-templating logic, such as validation, properties, utility methods, etc. can be placed in the partial .cs/.vb file. This helps to reduce complexity of .tt files by isolating “presentation” code that requires text templating in its own file where it is not mixed with the class definition and model access code.

Despite of what their name implies, Preprocessed Generator project items are not .tt files. A traditional Generator is typically defined in a single class feature block of a .tt file, does not use any text templating features and uses one or more traditional Templates to generate output files. Similarly, a Preprocessed Generator is a simple .cs/.vb file that contains a Generator class definition and uses one or more Preprocessed Templates to generate output files. Here is what it looks like.

C#

namespace ClassLibrary1
{
    using System;
    using T4Toolbox;

    public class Generator1 : Generator
    {
        protected override void RunCore()
        {

        }

        protected override void Validate()
        {
            this.Warning("Generator properties have not been validated");
        }
    }
}

Visual Basic

Imports T4Toolbox

Public Class Generator1
    Inherits Generator

    Protected Overrides Sub RunCore()

    End Sub

    Protected Overrides Sub Validate()
        Me.Warning("Generator properties have not been validated")
    End Sub

End Class

Just like in a traditional generator, you would typically define several properties and override RunCore and Validate methods.

T4Toolbox.Template Class Has a Breaking Change

In order to be compatible with template preprocessing in Visual Studio 2010, interface and implementation of the T4Toolbox.Template class had to change. TransformText method used to have a sealed implementation that called the abstract RenderCore method you had to override in the descendants. Now the TransformText method itself is abstract and serves the purpose of RenderCore which has been removed.

This is a breaking change for any existing traditional Template classes you may have. Here is an example of a Template implementation you had before.

C#

<#+
public class Template2 : Template
{
    public override void RenderCore()
    {
        // Text blocks and code generation logic
        // …
    }
}
#>

Visual Basic

<#+
Public Class Template2
    Inherits Template

    Public Overrides Sub RenderCore()
        ‘ Text blocks and code generation logic
        ‘ …
    End Sub

End Class
#>

And here is an example of a Template implementation you have to have now.

C#

<#+
public class Template2 : Template
{
    public override string TransformText()
    {
        // Text blocks and code generation logic
        // …
        return this.GenerationEnvironment.ToString();
    }
}
#>

Visual Basic

<#+
Public Class Template2
    Inherits Template

    Public Overrides Function TransformText() As String
        ‘ Text blocks and code generation logic
        ‘ …
        Return Me.GenerationEnvironment.ToString()
    End Function

End Class
#>

In order to be compatible with the new version of T4 Toolbox, all existing Template implementations need to be modified to a) replace each RenderCore method override with a TransformText method override and b) return GenerationEnvironment.ToString() at the end of the method.

Template class now provides a new method called Transform, which returns a string and serves the same purpose as TransformText implementation served previously. Just like TransformText, this method is not intended to be called directly by code in your Generator or Script code, where you would normally call Render or RenderToFile. You would only call TransformText (and now Transform) method in unit testing code to validate code generation results without creating the output files. If you have any code that calls the now abstract TransformText method, change it to call the new Transform instead.

Introducing this breaking change was not an easy decision. It was necessary to “unseal” implementation of the TransformText in order to make T4 Toolbox compatible with TextTemplatingFilePreprocessor in Visual Studio 2010 which assumes that this method is abstract. Although it would have been possible to keep backward compatibility, it would result in confusing design and implementations working differently in traditional and preprocessed templates. Taking into consideration the effort required to maintain this going forward, we made a decision to go ahead and convert existing templates now and leave this change behind.

The changes required to upgrade existing code are straightforward. You can use the “Find in Files” feature in Visual Studio to find all instances of RenderCore methods that need to be converted to TransformText. This is a manual change as it requires you to add a return statement at the end of each method. Replacing calls to TransformText method with Transform can be safely accomplished using the “Find and Replace” across multiple files. Including code changes, running unit tests and resolving errors, this change took less than an hour with the existing code generators in T4 toolbox itself.

Under the Hood

In order to make T4 Toolbox compatible with both Visual Studio 2008 and Visual Studio 2010, we now have to separate assemblies: T4Toolbox.dll, a .NET 3.5 assembly which is used in Visual Studio 2008, and T4Toolbox.10.0.dll, a .NET 4.0 assembly which is used in Visual Studio 2010. Introducing a separate new binary was required because Microsoft.VisualStudio.TextTemplating assembly in Visual Studio 2010 has been renamed to Microsoft.VisualStudio.TextTemplating.10.0, which means that TextTransformation base class in Visual Studio 2010 is now physically different than the same class in Visual Studio 2008, forcing you to have separate binaries for descendants.

Producing a .NET 4.0 version of the T4Toolbox assembly required a complete overhaul of the build system. The project files had to be upgraded to Visual Studio 2010. To keep complexity under control and have a single set of project files, they had to be hacked to generate an assembly with different target .NET framework versions depending on build configuration. This had to be done directly in the project files because target framework is exposed as a project-level setting by Visual Studio 2010.

Microsoft SDC tasks don’t appear to support MSBuild 4.0 at this time and generation of Visual Studio project item templates had to be changed to use MSBuild extensions in the DSL SDK.

Because of these changes, the previously published instructions on how to get and compile T4 Toolbox source code are now incorrect. I will not go back and update these instructions until we have a release version of Visual Studio 2010 and things settle down a bit.

©2010 Oleg Sych. All Rights Reserved.

During the first step, the engine preprocesses the template: it parses the processing instructions, text and code blocks, generates a concrete TextTransformation class, and compiles it into a .NET assembly. During the second step, T4 engine creates an instance of the GeneratedTextTransformation class, calls its TransformText method and saves the string it returns to the output file.

Visual Studio 2010 also allows you to preprocess the template at design time, when author of the code generator creates the template itself. At run time, when a developer is using template to generate output code, the hosting application simply creates an instance of the precompiled GeneratedTextTransformation and uses it to generate output as usual. Because preprocessed templates have no hard-coded dependency on Visual Studio, any application can host preprocessed templates, as long as it provides appropriate mechanism for saving or presenting the generated output to the user.

At the time of this writing, no official information is available about preprocessed text templates on MSDN. Gareth Jones and Pablo Galiano published useful information on their blogs (see the References section below). This article is meant to fill the gaps and serve as a comprehensive overview and reference of the functionality supported by preprocessed templates.

Creating a Preprocessed Template

• Create a new C# or Visual Basic project in Visual Studio 2010.
• Select Project -> Add New Item from the main menu

• In the Add New Item dialog, select Preprocessed Text Template item, enter appropriate name and click Add button.
• In Solution Explorer, you should now see the new .tt file and the preprocessed template file it generates. To see the preprocessed template file in a Visual Basic project, make sure to click the Show All Files button in the toolbar of Solution Explorer.

Note that Custom Tool this .tt file was associated with is TextTemplatingFilePreprocessor  instead of the traditional TextTemplatingFileGenerator. Instead of the output file, which would be generated by the traditional template, for the preprocessed template, we the actual source code of the template itself added to the project. This preprocessed template class is compiled as part of the project that contains the .tt file.

• Modify the .tt file to look like so

C#

<#@ template language="C#" #>
Hello World

Visual Basic

<#@ template language="VB" #>
Hello World

• Check the preprocessed template (PreprocessedTextTemplate.vb or .cs) which should look similar to this:

C#

Visual Basic

As you can see, the text, statement, expression and class feature blocks defined in the .tt file become a part of the preprocessed template class just like in a traditional template. However, there are several interesting points to note.

First of all, notice that name of the generated template class is based on the name of the .tt file. In our example, PreprocessedTextTemplate.tt produced class name PreprocessedTextTemplate. In a traditional template, this name would be hard-coded GeneratedTextTransformation. Having file name determine the class name allows us to give template classes meaningful names and organize them in class libraries.

The namespace in which the class is generated is based on the Custom Tool Namespace property of the .tt file. In Visual Basic, this property is My.Templates by default which is what you see generated in the code above. In C# this property is empty by default, causing TextTemplatingFilePreprocessor to construct the namespace based on the default project namespace and location of the .tt file in it. In Visual Basic, clearing the Custom Tool Namespace results in template class being generated in the root namespace of the project. This is also different from traditional templates, which use a hardcoded namespace with a randomly generated suffix, and also allows us to better organize the precompiled templates as part of class libraries.

And last, the preprocessed template class is generated as partial. You can further extend it by adding another partial .cs or .vb file to the project. This allows you to place methods and properties that don’t require templating in a regular source file instead of class feature blocks you would otherwise have to use in a traditional template. Out of the box, Visual Studio offers better debugging experience, IntelliSense and color syntax highlighting for regular source files. However, these differences are reduced only to debugging if you are using the T4 Editor to create your templates.

Using a Preprocessed Template

Executing a preprocessed template is a simple matter of creating an instance of the preprocessed template class and calling its TransformText method. As Pablo Galiano describes here, this can be done in any application. Here we will review how preprocessed templates can be used for code generation in a traditional T4 template, hosted and executed by Visual Studio.

• Add a traditional text template to the project

• Modify it to look like this, using full path to the assembly that contains the precompiled template and its namespace.

C#

<#@ template language="C#" #>
<#@ output extension="txt" #>
<#@ assembly name="C:\...\bin\Debug\CsTemplate.dll" #>
<#@ import namespace="CsTemplate" #>
<#
    PreprocessedTextTemplate t = new PreprocessedTextTemplate();
    this.Write(t.TransformText());
#>

Visual Basic

<#@ template language="VB" #>
<#@ output extension="txt" #>
<#@ assembly name="C:\...\bin\Debug\VbTemplate.dll" #>
<#@ import namespace="VbTemplate.My.Templates" #>
<#
    Dim t As PreprocessedTextTemplate = New PreprocessedTextTemplate()
    Me.Write(t.TransformText())
#>

As you can see, in this traditional template, we are creating a new instance of the precompiled template and writing the code it generates to the standard output file, TextTemplate.txt in this case.

• Check the output file, which should look like this.

Hello World

For simplicity of this example, we have both the preprocessed template and the traditional template using it in the same project. In a real-world scenario, you will typically have a preprocessed template in a separate assembly, which you compile and distribute to your development team for use in other projects.

Differences Between Traditional and Preprocessed Templates

There are several important differences between traditional and preprocessed templates.

Dependency on Microsoft.VisualStudio.TextTemplating assembly

Traditional T4 templates have a dependency on Microsoft.VisualStudio.TextTemplating assembly, which is installed as part of Visual Studio and cannot be redistributed. Preprocessed templates don’t hard-code this dependency and can be used to generate code from a custom hosting application. In the example above, you can see that all code required to execute the preprocessed template is generated as part of its definition and doesn’t rely on any base classes. However, when preprocessed templates are intended for use only within Visual Studio, we can avoid having multiple redundant implementations of the same properties and methods in each template class by using Inherits parameter of the Template directive to specify the standard TextTransformation as the base class.

Template Directive

Language parameter is partially supported. The precompiled template is generated in the specified language, however if template language doesn’t match the language of the project it belongs to, the generated template file will not be compiled. If this happens, the Build Action for the output file is automatically changed to Content.

Debug parameter appears to have no effect on preprocessed templates. Template author is responsible for setting debugging options for the project which contains the preprocessed .tt file.

Inherits parameter is supported and significantly changes the way preprocessed templates are generated. When this parameter is not present, the preprocessed template class will be generated without base class and will have its own implementations of all standard methods inherited by traditional templates from the TextTransformation class. When this parameter is present, the preprocessed template class will be generated with the specified base class and without the standard methods. Template author is responsible for specifying a base class that either inherits from TextTransformation, or provides compatible properties and methods.

Hostspecific parameter is partially supported. The preprocessed template class is generated with the appropriate Host property, however, this introduces compile-time dependency on Microsoft.VisualStudio.TextTemplating assembly. Template author is responsible for adding a reference to this assembly to the project that contains the preprocessed template class. The hosting application is responsible for providing a value for this property at run time.

Culture parameter appears to be fully supported.

Output Directive

The <#@ output #> directive appears to have no effect on preprocessed templates. No error is generated when a preprocessed template contains this directive. The hosting application is responsible for changing the extension of the output file.

Assembly Directive

The <#@ assembly #> directive appears to have no effect on preprocessed templates. No errors is generated when a preprocessed template contains this directive. Template author is responsible for adding appropriate assembly references to the project that contains the .tt file.

Import Directive

The <#@ import #> directive appears to be fully supported by the preprocessed templates. Appropriate using (C#) and imports (Visual Basic) statements are added to the generated template class.

Include Directive

The <#@ include #> directive appears to be fully supported by the preprocessed templates. Template blocks of the included files are merged appropriately in the generated template class.

Custom Directives

Custom directives, such as <#@ xsd #> and <#@ property #> appear to be fully supported by the preprocessed templates. The custom directive processors are invoked appropriately and any additional code they produce is merged with the template code in the generated template class.

References

• DSL 2010 Feature Dives- T4 Preprocessing - Part One - Rationale by Gareth Jones
• DSL 2010 Feature Dives- T4 Preprocessing - Part Two - Basic Design by Gareth Jones
• VS10 Beta 1 / T4 Preprocessing part 1 by Pablo Galiano
• VS10 Beta 1 / T4 Preprocessing part 2 by Pablo Galiano
• Why is a Preprocessed Template the Coolest Thing since Sliced Bread by Kathleen Dollard

Download

• Source code, C#
• Source code, Visual Basic 

©2010 Oleg Sych. All Rights Reserved.

Simplicity and ease of use makes T4 a better code generation tool for application developers, who typically don’t care about generating code in different languages and don’t have time to do it. On the other hand, ability to support multiple output languages makes CodeDOM a better code generation platform for tool developers, who have to support multiple output languages and don’t want to maintain several different code generators that generate the same thing in different languages.

During the past year, we have seen T4 getting adopted by an increasing number of tool developers inside Microsoft, most importantly by the ASP.NET MVC and ADO.NET EF teams. This is great news for application developers, who can now generate their code their way. Or is it? What about tool developers who now have to supply two code generators (T4/C# and T4/VB) instead of one? Actually, make that three code generators - they still have to supply and maintain the original CodeDOM-based generators to support other .NET languages. It is only a matter of time before this added complexity will show its ugly face in the form of quality problems, missing features, missing output languages, etc. In the end, the application developers might just get what they wished for - completely custom code generators that are their to implement and maintain.

Consider the following template from the previous article - Generating Code from DSL Models. This template (ClassTemplate) generates a class declaration based on a definition in a Class Diagram DSL (ModelClass). If you are using T4 Toolbox, you have already recognized the base class, Template, which inherits from TextTranformation. We could have also used pure T4 and have ClassTemplate inherit from TextTransformation directly.

C#

<#+
public class ClassTemplate : Template
{
    public ModelClass ModelClass { get; set; }

    public override string TransformText()
    {
#>
namespace <#= TransformationContext.DefaultNamespace #>
{
    using System;
    using System.Collections.Generic;

    public class <#= this.ModelClass.Name #>
    {
<#+
        foreach (ModelAttribute attribute in this.ModelClass.Attributes)
        {
#>
        public <#= attribute.Type #> <#= attribute.Name #> { get; set; }
<#+
        }
#>
    }
}
<#+
        return this.GenerationEnvironment.ToString();
    }
}
#>

Visual Basic

<#+
Public Class ClassTemplate
    Inherits Template

    Public ModelClass As ModelClass

    Public Overrides Function TransformText()
#>
Imports System
Imports System.Collections.Generic

Namespace <#= TransformationContext.DefaultNamespace #>
    Public Class <#= Me.ModelClass.Name #>
<#+
        For Each attribute As ModelAttribute in Me.ModelClass.Attributes
#>
        Public <#= attribute.Name #> As <#= attribute.Type #>
<#+
        Next
#>
    End Class
End Namespace
<#+
        Return Me.GenerationEnvironment.ToString()
    End Function
End Class
#>

Keep in mind that this template is simple only because it’s a sample. A real-world code generator that provides actual value, such as the LINQ to SQL entity class generator, can easily be thousands lines long. As a tool developer on the T4 Toolbox team, I don’t want to translate the LINQ to SQL template from C# to Visual Basic only so that I have to then maintain both. I would prefer to have a single code base to develop and maintain, but as a tool developer, I feel that I should not limit the audience of my tool just to C# developers.

Do we have to choose between the needs of application developers and the needs of tool developers? Is it possible for tool developers to use CodeDOM to develop code generators and for application developers to use T4 to customize and extend them? Well… YES! Thanks for asking!

How Would That Work?

A Tool developer would implement the ClassTemplate using CodeDOM and encapsulate code that generates individual type members, such as fields or properties, in virtual methods, such as RenderProperty.

C#

public class ClassTemplate : Template
{
  // …
  protected virtual void RenderProperty(ModelAttribute attribute)
  {
    CodeMemberProperty property = new CodeMemberProperty();
    property.Attributes = MemberAttributes.Public | MemberAttributes.Final;
    property.Type = new CodeTypeReference(attribute.Type);
    property.Name = this.LanguageProvider.CreateEscapedIdentifier(attribute.Name);

    property.GetStatements.Add(
      new CodeMethodReturnStatement(
        new CodeFieldReferenceExpression(
          new CodeThisReferenceExpression(),
          this.FieldName(attribute.Name))));

    property.SetStatements.Add(
      new CodeAssignStatement(
        new CodeFieldReferenceExpression(
          new CodeThisReferenceExpression(),
          this.FieldName(attribute.Name)),
        new CodePropertySetValueReferenceExpression()));

    this.LanguageProvider.GenerateCodeFromMember(
      property, this.generationWriter, null);
  }
  // …
}

There is no Visual Basic example of this code, because in this case, the tool developer chose to use C# as the programming language.

An Application developer would override the virtual method in a T4 template and use text templating instead of CodeDOM. The code below illustrates how the code generating properties could be extended to generate DataMemberAttribute declarations and make the properties serializable by WCF.

C#

<#+
    class T4ClassTemplate : ClassTemplate
    {
        protected override void RenderProperty(ModelAttribute attribute)
        {
#>
[DataMember]
<#+
            base.RenderProperty(attribute);
        }
    }
#>

Visual Basic

<#+
    Class T4ClassTemplate
        Inherits ClassTemplate

        Protected Overrides Sub RenderProperty(attribute As ModelAttribute)
#>
<DataMember> _
<#+
            MyBase.RenderProperty(attribute)
        End Sub
    End Class
#>

Note that the application developer is using T4 text templating features to extend a code generator written with CodeDOM. She did not have to reimplement the entire code generator in T4, only modify the logic she was interested in changing. In this example, she has extended the logic that generates property code using her language of choice - either Visual Basic or C#.

What’s the Trick?

Notice that our RenderProperty method uses GenerateCodeFromMember method to convert property declaration from a CodeMemberProperty object to text in the GenerationEnvironment, a StringBuilder object that accumulates output generated by our Template. This is why we can override the RenderProperty method in T4 and use text templating to change the generated code.

The rest of the magic is happening in the TransformText method, which builds the CodeDOM graph of the class we are generating using CodeSnippetTypeMember objects instead of the CodeMemberProperty, CodeMemberField and CodeMemberMethod objects we would use normally. The text for the CodeSnippetTypeMember objects is extracted from the GenerationEnvironment.

public class ClassTemplate : Template
{
  public CodeDomProvider LanguageProvider = CodeDomProvider.CreateProvider("CSharp");
  private StringWriter generationWriter;
  private CodeGeneratorOptions options;

  public ModelClass ModelClass { get; set; }

  public ClassTemplate()
  {
    this.generationWriter = new StringWriter(this.GenerationEnvironment);
  }

  public override string TransformText()
  {
    CodeNamespace @namespace = new CodeNamespace(TransformationContext.DefaultNamespace);
    @namespace.Imports.Add(new CodeNamespaceImport("System"));
    @namespace.Imports.Add(new CodeNamespaceImport("System.Collections.Generic"));

    CodeTypeDeclaration @class = new CodeTypeDeclaration(this.ModelClass.Name);
    @class.TypeAttributes = TypeAttributes.Public;
    @namespace.Types.Add(@class);

    foreach (ModelAttribute attribute in this.ModelClass.Attributes)
    {
       this.RenderField(attribute);
       @class.Members.Add(new CodeSnippetTypeMember(this.GenerationEnvironment.ToString()));
       this.GenerationEnvironment.Length = 0;
    }

    foreach (ModelAttribute attribute in this.ModelClass.Attributes)
    {
      this.RenderProperty(attribute);
      @class.Members.Add(new CodeSnippetTypeMember(this.GenerationEnvironment.ToString()));
      this.GenerationEnvironment.Length = 0;
    }

    this.LanguageProvider.GenerateCodeFromNamespace(@namespace, this.generationWriter, null);
    return this.GenerationEnvironment.ToString();
  }
}

In other words, to generate a particular type member, such as property, we a) build it as a CodeDOM graph; b) convert it to text stored in GenerationEnvironment of the Template; c) extract text from GenerationEnvironment and wrap it in a CodeSnippetTypeMember object; and d) finish building the CodeDOM graph of the type using code snippet objects. Because steps a) and b) occur in a virtual method, we can override it using pure text templating functionality of T4.

What’s the Catch?

Unfortunately, CodeDOM does not provide code snippet descendants for all types. In particular, there is no CodeSnippetTypeDeclaration descendant for CodeTypeDeclaration. Therefore, it is not possible to extract generation of the class declaration from the TransformText method into a virtual method similar to RenderProperty that it could be overridden using T4. Specifically, it would not be possible to extend the code generator in this example and use T4 text templating to generate DataContractAttribute declaration for the class. We can provide an event or a virtual method in ClassTemplate that would allow T4ClassTemplate to access and modify the CodeTypeDeclaration before the output code is generated from it, however T4ClassTemplate would need to manipulate it in the CodeDOM form. It is very much possible to do in T4 code, but would not be as easy as using the text templating.

So What Are You Saying?

As long as a CodeDOM-based generator provides adequate extensibility points, such as virtual methods and/or events, the application developers can extend or replace parts of its logic in T4 without having to reimplement or maintain the entire generator. Although this approach is significantly more complex than creating a T4-based code generator for a single output language, it is comparable in size and complexity to having multiple T4-based generators, producing the same code in different output languages. As the size and complexity of the generated code grows, maintaining several T4-based generators in different .NET languages becomes more complex than maintaining a single CodeDOM-based generator. This approach is a compromise between the needs of application developers, who want to customize generated code using T4 and their language of choice, and tool developers, who want to maintain a single code base.

Download

Source code, C# and Visual Basic. Follow instructions in the previous article to compile and run it.

©2010 Oleg Sych. All Rights Reserved.

Domain-Specific Language Tools

As the name implies, a Domain-Specific Language (DSL) is a programming language dedicated to a particular problem domain. Although we usually associate the term programming language with a textual language, such as C# and Visual Basic, DSLs also take a form of graphical modeling languages, such as UML diagrams, or Excel spreadsheets. Visual Studio SDK includes Domain-Specific Language Tools, which allow you to build DSLs of the graphical variety. Shown below, is a Class Designer built using DSL tools and included as a sample in the Visual Studio SDK.

This sample DSL allows you to model a set of classes in a graphical design environment, similar to a UML diagrams in Visio. The model it creates is stored in XML format.

The Class Designer example also includes T4 templates that you can use as a starting point to generate code from this model. These T4 templates rely on a custom directive processor called ClassDiagrams to make the model data available for code generation. The directive processor ensures that metadata assembly is referenced when the template is compiled and generates a special property called ModelRoot which provides access to the metadata loaded from the model file - Sample.classes.

C#

<#
/***************************************************************************
    Copyright (c) Microsoft Corporation. All rights reserved.
    This code is licensed under the Visual Studio SDK license terms.

    THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
    ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
    IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
    PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.

***************************************************************************/
#>
<#@ template inherits="ModelingTextTransformation" #>
<#@ output extension=".txt" #>
<#@ ClassDiagrams processor="ClassDiagramsDirectiveProcessor"
  requires="fileName=’Sample.classes’"  #>

Report template

<#= this.ModelRoot.Name #>

<#
  // When you change the DSL Definition, some of the code below may not work.

  foreach (ModelType type in this.ModelRoot.Types)
  {
#>
    <#= type.GetType().Name #> <#= type.Name #>
<#
  }
#>

Visual Basic

<#
REM    Copyright (c) Microsoft Corporation. All rights reserved.
REM    This code is licensed under the Visual Studio SDK license terms.
REM
REM    THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
REM    ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
REM    IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
REM    PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.
#>
<#@ template inherits="ModelingTextTransformation"language="VB" #>
<#@ output extension=".txt" #>
<#@ ClassDiagrams processor="ClassDiagramsDirectiveProcessor"
  requires="fileName=’Sample.classes’"  #>

Report template

<#= Me.ModelRoot.Name #>

<#
  REM When you change the DSL Definition, some of the code below may not work.

  For Each type As ModelType In Me.ModelRoot.Types
#>
    <#= type.GetType().Name #> <#= type.Name #>
<#
  Next
#>

The output produced by these templates is shown below. In a real-world scenario, you would probably generate something more useful, such as C# or Visual Basic class declarations or, perhaps, SQL table scripts.

Report template

T4Toolbox.Examples.OrderEntry

    ModelClass Customer
    ModelClass Address
    ModelClass Product
    ModelClass Supplier
    ModelClass Order

Because the DSL samples in Visual Studio SDK use standard T4 templates, they can only produce a single output file per template. As a result, you are either limited to generating all artifacts in a single large file, or having multiple templates to generate smaller files. Luckily, T4 Toolbox can be used to generate multiple files from DSL models as we will see later in this article.

Domain-Specific Language Implementation

Before we take a closer look at generating code, we will review some of the basics you will need to understand in order to follow discussion of code generation. For complete details, please refer to the Visual Studio SDK and Domain-Specific Development with Visual Studio DSL Tools written Gareth Jones and other people who lead creation of this technology at Microsoft.

The best way to follow this discussion is by opening the ClassDiagrams example from Visual Studio SDK.

The Class Designer DSL itself is also defined by a DSL model. The screenshot below shows a fragment of its definition located in the DslDefinition.dsl file inside of the Dsl project.

The diagramming notation is similar enough to UML class diagrams for you to be able to interpret it without reading a book on DSLs. As you can see, the sample DSL has a ModelRoot that contains a collection called Types of ModelType objects. ModelClass inherits from ModelType and stores information about a single class, including its collections of Attributes and Operations. In addition to the domain types, like ModelRoot and ModelClass, the DSL definition also includes diagram elements, such as ClassShape and AssociationConnector, which are not shown here.

The Dsl project contains most of the code required to implement the DSL itself and its design environment in Visual Studio. The GeneratedCode folder contains a set of T4 templates that generate most of the required code. In particular, it generates strongly-typed .NET code for accessing the resulting DSL models programmatically. Here is the template called DomainClasses.tt that generates metadata classes for the sample Class DSL.

<#
/***************************************************************************
    Copyright (c) Microsoft Corporation. All rights reserved.
    This code is licensed under the Visual Studio SDK license terms.

    THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
    ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
    IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
    PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.

***************************************************************************/
#>
<#@ include file="Dsl\DomainClassCodeGenerator.tt" #>
<#@ Dsl processor="DslDirectiveProcessor"
  requires="fileName=’..\DslDefinition.dsl’" #>

The screenshot below shows an extract from the code this template produces. The ModelClass shown in it is generated from the ModelClass element defined in the sample Class DSL diagram above. An instance of the ModelClass represents a particular class defined in the class model, such as Customer or Product shown in the beginning of this article.

ModelRoot, ModelType, ModelClass, ModelAttribute and other classes in this file are what our code generator will use to access model definition created by the Class Designer.

Generating Code From DSL Models

For our DSL code generator, we will use the standard T4 Toolbox design pattern based on Template and Generator classes.

C#

<#@ template language="C#v3.5" hostspecific="True" debug="True" 

    inherits="ModelingTextTransformation" #>
<#@ output extension="txt" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="ClassGeneratorCS.tt" #>
<#@ include file="ClassTemplateCS.tt" #>
<#@ ClassDiagrams processor="ClassDiagramsDirectiveProcessor" 

    requires="fileName=’Sample.classes’"  #>
<#
    ClassGenerator generator = new ClassGenerator();
    generator.ModelRoot = this.ModelRoot;
    generator.Run();
#>

Visual Basic

<#@ template language="VBv3.5" hostspecific="True" debug="True" 

    inherits="ModelingTextTransformation" #>
<#@ output extension="txt" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="ClassGeneratorVb.tt" #>
<#@ include file="ClassTemplateVb.tt" #>
<#@ ClassDiagrams processor="ClassDiagramsDirectiveProcessor" 

    requires="fileName=’Sample.classes’"  #>
<#
    Dim generator As ClassGenerator = New ClassGenerator()
    generator.ModelRoot = Me.ModelRoot
    generator.Run()
#>

The sample generator consists of two separate classes. ClassTemplate, defined in ClassTemplateCs.tt or ClassTemplateVb.tt, takes ModelClass and generates a class declaration. ClassGenerator, defined in ClassGeneratorCs.tt or ClassGeneratorVb.tt, takes ModelRoot as a parameter, iterates through its list of Types and renders ClassTemplate for each ModelClass it finds. There is nothing special about design or implementation of ClassTemplate and ClassGenerator, except for the ModelClass and ModelRoot types they use as parameters for code generation. These types are provided by the Class Diagrams DSL.

This code generator produces class declarations for types defined in Sample.classes in separate source files.

C#

namespace T4ToolboxDslCs
{
    using System;
    using System.Collections.Generic;

    public class Address
    {
        public String Street { get; set; }
        public String City { get; set; }
        public String State { get; set; }
        public String Zip { get; set; }
    }
}

Visual Basic

Imports System
Imports System.Collections.Generic

Namespace T4ToolboxDsl
    Public Class Address
        Public Street As String
        Public City As String
        Public State As String
        Public Zip As String
    End Class
End Namespace

The main code generation file, SampleCs.tt or SampleVb.tt depending on your language preference, does have several important differences from a typical T4 Toolbox template. First important difference is the <#@ ClassDiagrams #> custom directive placed in the beginning of the template. This custom directive has a requires parameter that expects a value in form fileName=’<FilePath>’, where <FilePath> is a relative path to the DSL model file. The ClassDiagrams directive processor is provided by the Class Diagrams DSL and registered in the experimental registry hive of Visual Studio when you run the Dsl project in the Visual Studio SDK sample. Once the DSL is deployed this directive processor would be registered under standard Visual Studio registry hive.

[HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0Exp\Configuration\TextTemplating\DirectiveProcessors\ClassDiagramsDirectiveProcessor]
@="A directive processor that provides access to ClassDiagrams files"
"Class"="Microsoft.Example.ClassDiagrams.ClassDiagramsDirectiveProcessor"
"CodeBase"="file:///C:/Users/you/Desktop/DslPackage/bin/Debug/Microsoft.Example.ClassDiagrams.Dsl.DLL"

Second important difference is the use of ModelingTextTransformation in the inherits parameter of the template directive. This base class provides additional methods, such as AddDomainModel and ConvertModelRelativePathToTemplateRelativePath, which are required for the code generated by the ClassDiagrams directive processor to be compiled correctly during template transformation.

Running Sample Source Code

Complete source code is available for download in the end this article. It is based on the ClassDiagrams sample from Visual Studio SDK. In order to see the code generation in action, you will need to start debugging the Dsl project in Example.ClassDiagrams solution.

This will start a second instance of Visual Studio in the experimental hive and load Debugging solution. This solution contains several projects. Debugging project came in Visual Studio SDK with this sample and contains standard T4 templates shown at the beginning of this article.

T4ToolboxDslCs and T4ToolboxDslVb are the C# and Visual Basic projects that contain code generators based on the T4 Toolbox. SampleCs.tt and SampleVb.tt are the actual code generators that produce class declarations in separate source files. Remember that you need to click the Show All Files button in the toolbar of Solution Explorer in order to see the output files in Visual Basic projects.

Conclusion

Graphical Domain-Specific Languages are a very powerful tool that can be used to build complete solutions for modeling and generating code for complex problem domains. DSLs are typically developed by Microsoft and other companies who specialize in building development tools. Due to inherent complexity of designing a graphical language and implementing a complete modeling environment integrated in Visual Studio IDE,  DSL development is not feasible in scope of a typical application development project. The investment required to implement and support a viable DSL can only be recouped on some of the largest application development projects that involve dozens of people and last for several years.

These larger projects, where DSL development is justified, typically require generating large volumes of code. Due to the T4 limitation of generating a single output file per template, developers are forced to generate multiple classes in each output file and use multiple template files to generate code from a single model to keep the size of output files manageable. A great example of this problem is the implementation of a DSL itself, which has a single model definition file and multiple code generation templates files. Each of the generated files contains numerous class definitions, making them difficult to read and debug.

The traditional T4 templates are also impossible to extend without resorting to modifying their source code directly. On a large application development project, the team developing the DSL quickly becomes a bottleneck for customization requests. Alternatively, the teams using the DSL start customizing the main T4 templates which requires additional effort to maintain them over time as new versions of the DSL are released.

Although the issues around template extensibility and output file management not be difficult for authors of the DSL, the large number and size of the T4 templates required for non-trivial scenarios present a serious challenge for application developers using it. T4 Toolbox can help you address these problems. By implementing your DSL code generator as a set of Template and Generator classes as shown in this article, you can generate all required output files from a single T4 file and allow developers to extend it with the help of inheritance and encapsulation instead of modifying the main code generator directly.

Download

Source Code, C# and Visual Basic

Resources

• Domain-Specific Language Tools
• Domain-Specific Development with Visual Studio DSL Tools
• Creating reusable code generation templates
• Creating complex code generators
• Making code generators extensible
• Reusing code generators on multiple projects

©2010 Oleg Sych. All Rights Reserved.

Out of the box, TFS 2008 comes with two varieties of process guidance: MSF for Agile Software Development and MSF for CMMI Process Improvement. MSF Agile process was designed for smaller teams and teams that use agile development practices, while MSF CMMI process was designed for larger organizations, particularly those using CMMI (Capability Maturity Model Integration) approach, teams that require a higher degree of control and visibility over their projects.

You can learn more about MSF Agile and MSF CMMI by reading the process guidance that comes with TFS. If you don’t have access to a TFS project with the process flavor you are interested in, you can download it from MSDN using the links below. Simply open ProcessGuidance.html located in the Windows SharePoint Services\Process Guidance folder in the location where you extracted the project template files.

MSF Process Guidance is Only a Template

While MSF Agile and MSF CMMI templates can serve as good starting points for many organizations, there is rarely a perfect fit. The Agile template fits well most of the small- to mid-size IT organizations, however it uses terminology that is too specific to agile software development. Its term Scenario represents what is commonly referred as Service Request or Change Request; the term Bug refers to what is commonly known as Trouble Ticket or Defect Report. The CMMI template uses more appropriate terminology, but due to the large number of work-item types and roles, it is simply an overkill for most organizations.

Today, a typical MIS department usually has a small number of roles (such as Business Analyst, Project Manager, Developer, Database Administrator, Help Desk Technician) established and follows an existing process for managing the change requests and trouble tickets. This process may be based on paper or electronic forms, it may not be perfect, but it is already working. When this organization adopts TFS to improve their process, should they go ahead and adopt MSF Agile or MSF CMMI?

Adopting a new process approach (what you do) at the same time as adopting a new process management tool (how you do it) is a risky proposition. People are creatures of habit and require time to change either the process they follow or the tools they use. Imagine if all cars in a country had to be changed to use voice control instead of traditional steering (change what you do) and that people had to speak to their cars in Esperanto (change how you do it). In the resulting chaos, we can be certain that both the congressman who proposed this improvement and the president who didn’t veto the proposal will not be as popular anymore.

When adopting TFS, unless your IT organization is following the MSF Agile  or the MSF CMMI process to the letter, it is much safer to start by customizing the TFS process to match your own. Once your organization is comfortable using TFS as the new process management tool, you can begin adjusting the process itself using TFS. Microsoft provided extensive documentation (see links below) on how to customize the workflows, work items, reports and other features of TFS. However, customization of the process guidance received relatively little attention. Yet in the process of adopting TFS, providing matching guidance for a customized process is very important to ensure the overall success. Outdated guidance will cause confusion, mistakes, delays and frustration for the MIS department and its business partners. So how do we change it?

VSTS Process Guidance Generator

Microsoft used the Guidance Generator to produce MSF Agile and MSF CMMI guidance for TFS 2008. This tool replaced MSFWinBuild which was used to produce the guidance in TFS 2005. Unlike its predecessor, it uses plain XML file to define contents of the guidance and T4 templates to generate the actual collection of HTML files, included in TFS project templates as process guidance. The remainder of this article will show a typical set of steps you would need to perform to customize guidance for your TFS project.

Installing Guidance Generator

• Download latest version of the Guidance Generator (2.0.2.1 at the time of this writing) to your local hard drive.
• Unblock the ZIP archive you downloaded as shown on the right. This will ensure that the application will not run into any unexpected security errors.
• Extract files from the unblocked archive to your hard drive.

If you are running a 64-bit version of windows, you will need to use the CorFlags command line tool to make sure that the generator runs in 32-bit mode. This will help you to avoid errors that occur when the generator attempts to load 32-bit-only assemblies it requires.

• Use Command Prompt window to execute the following command in the bin\bin sub-folder at the location you extracted the files.

C:\GuidanceGenerator\bin\bin>corflags TextTemplatingMultiprocessor.exe /32BIT+
Microsoft (R) .NET Framework CorFlags Conversion Tool.  Version  3.5.21022.8
Copyright (c) Microsoft Corporation.  All rights reserved.

C:\GuidanceGenerator\bin\bin>

Building Process Guidance

Once you have the Guidance Generator installed, your first task is to pick a template (Agile or CMMI) that you will be using as a starting point in your customization. If you are customizing guidance for an existing TFS project, you will want to minimize the amount of editing, so pick the template that matches matches your project.

• Double-click the Run Tool.cmd file in root folder at the location you extracted the files to start the generator.   
• In the TemplateProcessor window, enter the following parameter values.

  Source DSL File:      agile-guidance.guidance

  Template Directory: HTML_Templates

  Target Directory:       agile-Generated\Process Guidance\Supporting Files

The source file provides content of the guidance. In this example, we are generating process guidance from the source file called agile-guidance.guidance to the output directory called agile-Generated. To generate CMMI guidance, we would need to change these values to cmmi-guidance.guidance and cmmi-Generated respectively.

• Click the Go button.

The generator runs for a while and you should see the a log displayed in its output window with information about the number of files that were generated. Once you see “Done”, you can open the generated guidance by double-clicking the ProcessGuidance.html located in the agile-Generated\Process Guidance folder. The actual output files are generated in the agile-Generated\Process Guidance\Supporting Files folder.

Customizing Guidance Content

The guidance content is located in the .guidance files - agile-guidance.guidance and cmmi-guidance.guidance. These are rather large XML files that define topics describing team member roles, work streams, activities, work products, work items, reports, glossary entries and more.

Limited information about the structure of this file can be found in the UserGuide - Guidance Generator.pdf located in the root folder where you extracted the Guidance Generator files downloaded from CodePlex. This information is by no means complete nor, due to its volume, would not be practical to describe in a single article. In one sentence, this structure can be summarized as this. There is a one-to-one relationship between the major xml elements and the generated HTML files, for example, BusinessAnalyst.htm file is generated from the XML element <Role Key=”BusinessAnalyst”>. If you want to change content for particular HTML file, find the XML element with matching Key and change its contents.

Source XML

Generated Output

Editing such a large and complex XML file is challenging. It was not meant to be edited by hand and the internal tool Microsoft created for editing it was not released to public. You may be tempted to use the XML schema file, GuidanceModelSchema.xsd, to enable IntelliSense and enhance your editing experience in Visual Studio, but unfortunately, it appears to be outdated and doesn’t match the supplied .guidance files.

Instead, you will need to frequently re-run the generator to verify that content you are modifying produces the HTML you expect. You can reduce the amount of time it takes to regenerate output by setting Template Files parameter to regenerate only a the particular type of file you are working on. For example, Business Analyst.htm is generated by the T4 template called Role.htm and you can regenerate only role files like so.

Customizing Guidance Output

Guidance Generator uses T4 templates located in the HTML_Templates directory to generate output HTML files (in the Supporting Files directory). There is one template file for each major type of XML element in the .guidance file (<Role>/Role.htm, <Activity>/Activity.htm). As you can see, the T4 template files are saved with .htm extension which might be confusing at first.

The final HTML files rely on additional web resources - JavaScript, Cascading Style Sheet and image files located, respectively, in the Code, CSS and Images sub-folders of the Supporting Files folder where the generated output files are saved. Each guidance (Agile and CMMI) has its own set of supporting files.

To change look and feel of the generated HTML, you can start by changing the CSS\msf.css style-sheet or changing the image files to display your company’s logo. However, if you need more substantial changes, you may have to modify the T4 templates that generate the HTML. For detailed information about T4, please see my previous article, Text Template Transformation Toolkit.

Publishing New Guidance

Once you finish changing the guidance, you can package it in a TFS process template, or simply upload it to the Process Guidance folder on your team portal.

Conclusion

TFS is a powerful collaboration platform for IT organizations. Although TFS comes with two pre-packaged process templates (MSF Agile and MSF CMMI), it is often beneficial to customize them to better fit the process already used by your organization. It is also important to keep documentation in sync with the process you are using and minimize confusion among the members of the immediate IT team as well as the business users. Guidance Generator is a powerful tool for customizing the built-in process guidance that comes with TFS. Although the Guidance Generator is not very easy to use, the interactive documentation system it produces is well designed, easy to use and would be challenging to replicate in hand-crafted HTML or Word files. In most cases you will want to start with an existing process guidance that most closely matches your current process and customizing it, hopefully by simply removing the MSF concepts you currently don’t use.

References

• MSF for Agile Software Development (TFS Project Template)
• MSF CMMI Process Improvement (TFS Project Template)
• How To: Customize a Process Template in Visual Studio Team Foundation Server
• Team Development with Visual Studio Team Foundation Server
• VSTS Process Guidance Generator
• Text Template Transformation Toolkit
• Team Foundation Administrators: Customizing Process Templates

©2010 Oleg Sych. All Rights Reserved.

T4 Toolbox now comes with a set of project item templates for Visual Basic. Here is what you will see when adding a new item to a Visual Basic project.

If you are familiar with these project item templates in C#, you will recognize the code in Visual Basic.

File

<#@ template language="VBv3.5" hostspecific="True" debug="True" #>
<#@ output extension="txt" #>
<#@ include file="T4Toolbox.tt" #>
<#
‘ <copyright file="File1.tt" company="Your Company">
‘  Copyright © Your Company. All Rights Reserved.
‘ </copyright>

#>

Template

<#+
‘ <copyright file="Template1.tt" company="Your Company">
‘  Copyright © Your Company. All Rights Reserved.
‘ </copyright>

Public Class Template1
    Inherits Template

    Public Overrides Function TransformText()    

        Return Me.GenerationEnvironment.ToString()
    End Function

End Class
#>

Generator

<#+
‘ <copyright file="Generator1.tt" company="Your Company">
‘  Copyright © Your Company. All Rights Reserved.
‘ </copyright>

Public Class Generator1
    Inherits Generator

    Protected Overrides Sub RunCore()    

    End Sub

End Class
#>

Entire functionality of the core framework of T4 Toolbox is now available to Visual Basic developers. I have updated the T4 tutorial series to provide examples in both C# and Visual Basic.

• Creating your first code generator
• Troubleshooting code generation errors
• Debugging code generation files
• Creating reusable code generation templates
• Creating complex code generators
• Reusing code generators on multiple projects
• Handling errors in code generators
• Unit testing code generators
• Making code generators extensible

Under the hood

In order to make Visual Basic support possible, the entire core framework, including TransformationContext, Template, Generator and other classes, was moved from .tt (Text Template) files to .cs (C#) files and is now compiled in the T4Toolbox.dll assembly. Some of the original extensions, such as initialization of the TransformationContext and TestRunner classes had to be reimplemented using custom directive processors.

Note that the set of templates available in Visual Basic is smaller than for C#. That is because ready-to-use code generators, such as AzMan wrapper, Enum SQL view and LINQ to SQL model are still implemented in C#. There are no plans to convert them to Visual Basic at this time.

Having the entire core framework code compiled as part of a normal C# class library project, we will finally be able to use Sandcastle and generate documentation for T4 Toolbox.

Known issues

Developers working primarily with C#, like myself, may be surprised to find that Visual Basic doesn’t display generated output files by default. You have to select the Show All Files button in toolbar of Solution Explorer in Visual Studio in order to see them.

A bigger issue is using Visual Basic in partial templates that contain code meant to be included and never used separately, such as Template and Generator classes. Without the template directive, current version of the T4 Editor assumes the language to be C# and displays a large number of errors for Visual Basic code. As a workaround, you can temporarily add a template directive to such file. To avoid warnings, it has to be removed once the file is included and compiled as part of another .tt file because T4 does not support multiple template directives.

This workaround is quite painful to use. To make this work right, the T4 editor needs to somehow infer the language from files without the template directive. One of the possible ways to do this is with file extensions. For example, partial C# text templates could use extension .ttcs and partial Visual Basic templates could use .ttvb. What do you think? Is there a better way to solve this?

©2010 Oleg Sych. All Rights Reserved.