T4 Architecture


Text templates run within the constraints established by the T4 architecture. Well designed templates take advantage of the benefits offered by T4 and sometimes need to work around its limitations.

Template Transformation Process

Here is a collaboration diagram that illustrates the template transformation process and its participants. Sections below provide a step by step description of the process. References to objects and steps from this diagram use italic font.

T4 Architecture

Custom Tool

Template transformation process begins in Visual Studio when you right-click a .tt file in Solution Explorer and select “Run Custom Tool” from the context menu. Custom Tool is a special COM object that implements IVsSingleFileGenerator interface, defined by Visual Studio. The T4 Custom Tool is implemented by a .NET class called TemplatedCodeGenerator and registered in Visual Studio settings under name TextTemplatingFileGenerator. Visual Studio locates the Custom Tool by its name, creates the COM object and calls its Generate method (step 1).

ITextTemplating Service

The custom tool uses IServiceProvider interface provided by Visual Studio to locate a service provided by T4, called ITextTemplating. This service is implemented in Microsoft.VisualStudio.TextTemplating.VSHost assembly by an internal class called TextTemplatingService. Once the service is located, the Custom Tool calls its ProcessTemplate method, passing it template contents and some other parameters (step 1.1).

Host

TextTemplatingService serves as a Host for the code generation Engine. While the Engine is responsible for code generation, the Host is responsible for providing all external resources it requires. The Host implements ITextTemplatingEngineHost interface defined by T4. During code generation, the Host passes this interface along with the template contents to the ProcessTemplate method of the Engine for code generation (step 1.2).

Engine

The T4 Engine is responsible for parsing the template (step 2), generating and compiling a GeneratedTextTransformation (steps 3, 4 and 5),  running it and returning output it produced to the Host (step 6). After parsing the template contents (step 2), the Engine processes T4 directives used in the template. While processing a built-in directives (step 3) the Engine calls the Host to perform the required action. For example, to process the include directive the Engine calls the LoadIncludeText method of the Host (step 3.1).

DirectiveProcessor

When processing custom directives (step 4), the Engine first calls the ResolveDirectiveProcessor of the Host (step 4.1) to determine the Type of DirectiveProcessor for each of them. Directive Processors work by adding code to the GeneratedTextTransformation. For example, property directive creates a property that can be used to pass a parameter value to the template from the outside. Once processors for all types of custom directives are resolved, the Engine instantiates and calls the ProcessDirective of an appropriate processor for each custom directive (step 4.2).

GeneratedTextTransformation

The Engine combines parsed template blocks and output of custom directives to produce source code for the GeneratedTextTransformation, a special class that inherits from TextTransformation class provided by T4 and represents a "compiled" version of the template (step 5).

The Engine calls the ResolveAssemblyReference method of the Host to locate all assemblies referenced by the template and required to compile it (step 5.1). It then calls the ProvideTemplatingAppDomain method of the Host to obtain the AppDomain where the code will be compiled (step 5.2). Visual Studio Host will create a new AppDomain called TemplatingAppDomain (step 5.3). If the TemplatingAppDomain already exists, Visual Studio Host will reuse it up to 25 times to improve performance.

After obtaining a TemplatingAppDomain, the Engine uses a CodeDomProvider (C# or Visual Basic, depending on the language specified in the template directive) to compile the the source code of the GeneratedTextTransformation in a temporary, in-memory assembly (step 5.4). This temporary assembly remains cached in the TemplatingAppDomain. If the same template is transformed again, the Engine will find find a cached assembly instead of generating a new one.

Once GeneratedTextTransformation is compiled, the Engine creates a new instance of this class and calls its TransformText method (step 6). This method returns output content generated by the template, which then goes back to the Engine, the Host, the Custom Tool and finally to the Visual Studio.

Analysis

Sections below highlight some of the aspects of T4 architecture. While some of them represent constraints that T4 architecture places on template design, others represent opportunities that may be overlooked by template developers.

T4 is designed to generate one output file per template

A close look at the Generate method of the IVsSingleFileGenerator interface reveals that a Custom Tool is responsible only for generating content. The custom tool returns generated content to Visual Studio, which is responsible for saving it to a file, adding the output file to the current project and placing it under source control. It’s a great design that allows custom tool to know very little about Visual Studio internals, however, it also limits T4 to producing a single output file per template. T4 templates need to use special workarounds to generate multiple output files from a single template.

ITextTemplating service can be used in Visual Studio add-ons

Use of ITextTemplating service is not limited just to the T4 Custom Tool. It is possible to use this service directly from a Visual Studio add-on. For example, a DSL designer could use this service to generate code directly from domain model using one or more standalone templates, without requiring users to add and manage .tt files in their projects.

Behavior of built-in directives can be changed by implementing a custom host

Keep in mind that the Engine handles built-in directives without creating directive processors. For example, it handles the output directive by calling the SetFileExtension and SetOutputEncoding methods. In order to change behavior of built-in directives you would need to create a custom T4 host.

Behavior of T4 directives varies between different hosts

Ability to create a custom T4 host is a double-edged sword. Three different T4 host implementations are available from Microsoft - Visual Studio host (described in this article), command line host and GAT host. Functionality provided by these hosts varies significantly and affects behavior of assembly, include and property directives, making it more difficult to reuse the same template with different hosts.

Custom directives can extend T4 template syntax

You can create a custom directive processor for your T4 templates and register it with Visual Studio Host. Unfortunately, command line host does not support the same registration mechanism, which may limit your ability to use custom directives. 

Assemblies referenced by T4 template become locked

Remember that Visual Studio Host reuses the templating AppDomain up to 25 times to improve performance. Unfortunately, this comes at a cost. Once the assembly containing GeneratedTextTransformation class is loaded in the templating AppDomain, the .NET Framework locks all of the assemblies it references. If your template uses one of your own assemblies, you will not be able to recompile it after running the template. This problem cannot be easily solved if your template needs run code from your assembly. However, if your template simply uses Reflection to generate code based on types in your assembly, you can work around this problem by using a different API, such as Introspection or CodeModel to access type metadata.

GeneratedTextTransformation runs in Visual Studio process

Ultimately, GeneratedTextTransformation runs in the Visual Studio process (devenv.exe), which hosts the TemplatingAppDomain. One of the side effects of this design is that debugging the compiled template requires you to attach a debugger to the Visual Studio itself. In other words, you typically need a second instance of Visual Studio to debug your template running inside the first one.

Another side effect of this design is that Visual Studio process defines the CurrentDirectory for the running GeneratedTextTransformation, which doesn’t match the directory where the template itself is located. This makes it more difficult to use relative paths referring to external files in T4 templates. As described in the next section, template authors need to use ResolvePath method to work around this problem.

Templates can access the Host

Templates can access the Host using the hostspecific parameter of the template directive. This parameter adds a Host property to the GeneratedTextTransformation class giving the template access to the instance of the ITextTemplatingEngineHost interface. This gives to the template author to use the ResolvePath method to convert relative paths to absolute paths. For example, AzMan template uses this approach to allow you to specify a relative path to the model (XML file that contains AzMan security definitions) used to generate a strongly-typed .NET wrapper. Without this ability, use of templates in team projects would be difficult because file paths would need to be hard-coded.

Templates can access Visual Studio services

Visual Studio Host implements IServiceProvider interface, which template can obtain by type-casting the instance of ITextTemplatingEngineHost available via its Host property. Having this interface instance, template can use its GetService method to access services provided by Visual Studio. For example, template can access DTE object and add multiple output files to a Visual Studio project.

About T4

T4 (Text Template Transformation Toolkit) is a template-based code generation engine. It is available in Visual Studio 2008 and as a download in DSL and GAT toolkits for Visual Studio 2005. T4 engine allows you to use ASP.NET-like template syntax to generate C#, T-SQL, XML or any other text files.

Additional information about T4 architecture is available in MSDN article Architecture of the Text Template Transformation Process. For more information about T4, check out my previous article.


Write a Comment

Take a moment to comment and tell us what you think. Some basic HTML is allowed for formatting.

Reader Comments

Most interesting topic. I am looking for feedback on the feasibility for leveraging a DSL of our legacy IDE in order to generate a C++ 9.0 analog of my agency’s legacy MFC “gray lady”. She is a complex combo of static and dynamic libraries accessing MFC, Win32 API and active X automation.

The good news (I think)is that I have been able to port and compile the legacy IDE’s SDK in a C++ 9.0 VS2008 project. As far as I can tell, this is a complete map of the visual, IDE outline and language layers which comprise the IDE. In addition, the legacy source libraries may be saved as text.

With your knowledge of this amazing T4 code generation capability, is there any hint of a path to .NET here, whether that be a full portage or some combination with the PInvoke/Marshalling capabilities in C++ 9?

Thanks for any insight you might provide!

Daniel

Daniel,

It is difficult to say, knowing so little about your system. You can definitely generate C++ code using T4. If the DSL of your legacy IDE is already used to generate MFC code and provides enough metadata to generate a significant portion of the system, you may be able to leverage T4 to generate equivalent .NET (C#, VB) code. Otherwise, porting existing source code to latest version of C++ and gradually introducing C++/CLI may be the best option.

Oleg

Oleg,

I first owe you the common courtesy of making time to dive into all the great material posted here and elsewhere. Perhaps then I will be able to engage in more substantive exhange. The one thing I DO know very well is the source code for our good ole’ MFC/MDI app.

More good news - the IDE comes with an alternative compliler. A cursory test run compiled about half of the source system’s functional code to native DLL, and also produced a non-human-readable native-compiled version of the IDE’s proprietary, outline-style include library file (*.APC).

Might that new DLL then be reflected back into new cpp9 source code? that would leave the remaning half to be targeted with the DSL/TT approach. I have not yet accomplished the creation of that “IDE DSL”.

I believe we are fortunate in that our amcient version of the legacy IDE (circa ‘98) actually precedes the introduction of COM/COM+ being wrapped into the proprietary platform.

Sp, in terms of the VS2008 DSL designer options, would my envisioned DSL of that object oriented, MFC based, outline-styled IDE be most akin to a (conceptual)component model? a class diagramer? Is a task flow model of the IDE’s run-time behavior called for? If this “IDE DSL” is a success, will that mean that the platform’s proprietary compiler can be “Put out to pasture? And finally, does the removal of licensing restrictions by microstoft on the MFC libraries present any novel opportunities to expedite/facilitate such a project, given that the sealed class templates made available to the coder within the legacy IDE are, to my eye at least, really just MFC instance factories?

I find that the idea of creating a visual language and debugger to model “another” ide puts me in mind of learning recursion years ago in college. Think about it TOO long and I start to feel a bit dizzy.. then I have to stand up and whistle for a while….

Thanks!
Dan

Hello Again!

I am finally back to revisit this absolute marvel of a resource after 3 years of “organizational gestation” where I work. Finally today the organizational decks have been cleared; the headlong fire-drill/death-march has ended; the phonograph needle lifted from the crazy music that has accompanied an unending round of musical deck chairs; although much flotsam and jetsam still abound, the rainbow has appeared, the dove presents an olive branch to all, and I look forward with great relish to the challenge ahead - applying T4 principals to porting a mature, robust enterprise application layer to .NET, TFS and VS.

Q: With the improvements in C++ 0x and VS2010 cpp language support since summer 2008, does your previously suggested path of [”porting existing source code to latest version of C++ and gradually introducing C++/CLI”] present EVEN LESS “resistance” NOW than that had we pursued it in August 2008? If so, how so? What improvements in T4 might apply to my problem domain?
Thanks!
Daniel

[…] I certainly don’t mean to dismiss T4 entirely. In particular, there are some 3rd-party Visual Studio plugins that make T4 templates much easier to work with, by providing syntax highlighting and intellisense among other features: Tangible T4 Editor and Clarius Visual T4 both offer similar feature sets and have a similar business model (a limited free version and a full-featured ‘pro’ version). Additionally, T4 is highly extensible; you can create your own text templating host and directive processors to control how your templates are compiled and run. For an excellent overview of the T4 architecture and many more T4 resources, see Oleg Sych’s blog. […]

[…] a working software solution. At first I used T4Toolbox and I the great resources available from Oleg Sych to manage generation of multiple files from one T4 template. But after some time, one reason being […]