Oleg Sych

While developing enterprise applications, development teams commonly face a challenge of ongoing development of new features and maintenance of the version already released to production. Consider a team that implements a new feature in April. The new feature goes through testing and gets released to production. The team begins working on the next new feature and starts actively changing code. Then in May, a serious bug is reported by users in production environment. The development team needs to make a hot fix but there is not enough time and not enough resources to put the hot fix through a full test cycle. How can the team make sure that the problem is fixed production without all the changes that were made to implement the new feature for next major version?

A proven approach to solving this problem is to take the snapshot of the source code that was used to compile the version of the application currently in production; make the changes required to fix the problem in this snapshot; build a new version that only contains the bug fixes; concentrate on testing the bug fixes; limit amount of the overall system testing based on available resources and impact of the changes; deploy the hot-fix build to production.

Extensive guidance exists on how to support this scenario using source control functionality in Team Foundation Server. In particular, Team Development with Visual Studio Team Foundation Server (TFS Guide) provides a comprehensive list of strategies in Chapter 5. TFS 2008 Branching Guide is a more recent set of specific guidelines, consistent with Chapter 5 of the TFS Guide. Here is a set of scenarios TFS Guide defines.

• Scenario 1 – No Branches. Your team works only from the main source tree. In this case, you do not create branches and you do not need isolation. This scenario is generally for small or medium size teams that do not require isolation for teams or for features, and do not need the isolation for releases.
• Scenario 2 – Branch for Release. Your team creates branches to support an ongoing release. This is the next most common case where you need to create a branch to stabilize for a release. In this case, your team creates a branch before release time to stabilize the release and then merges changes from the release branch back into the main source tree after the software is released.
• Scenario 3 – Branch for Maintenance. Your team creates a branch to maintain an old build. In this case, you create a branch for your maintenance efforts, so that you do not destabilize your current production builds. You may or may not merge changes from the maintenance branch back into the main tree. For example, you might work on a quick fix for a subset of customers that you do not want to include in the main build.
• Scenario 4 – Branch for Feature. Your team creates branches based on features. In this case, you create a development branch, perform work in the development branch, and then merge back into your main source tree. You can create separate branches for work on specific features to be completed in parallel.
• Scenario 5 – Branch for Team. You branch to isolate sub-teams so they can work without being subject to breaking changes, or can work in parallel towards unique milestones.

These scenarios provide solid advice that will scale to largest teams and projects. However, we need to keep in mind that there is an immediate overhead associated with multiple branches in the form of the initial learning curve every team member needs to go through as well as ongoing effort required to merge between branches. On a small team where developers work on one project at a time and projects have medium cycle durations (measured in a few months), the need to release hot fixes may arise only occasionally. In scenarios like this, branching approach may be an overkill, causing more problems than it solves. An alternative approach can be used to achieve the goal of releasing a hot fix without having to resort to branching and merging.

Solution

With Team Foundation Server, we can rely on workspaces to obtain the isolation required to work with a snapshot of the released source code. The idea is to get a labeled snapshot of source code of the production build; make the changes required to fix the problem; check in the modified files; apply a hot-fix label to the files in the workspace; and make Team Build get and build the hot fix version instead of latest. Let’s take a look at these steps in more detail.

Get source code released to production

Assuming that development team is already using Team Build, here is how one could get a snapshot of the production source code from source control.

• In Source Control Explorer, right click root folder of the team project and select Get Specific Version from the context menu.
• In the Get dialog, select Label as Version Type and click the […] button to find the label of the build that was released to production.

• In the Find Label dialog, select the name of your project in the Project drop-down list and click Find.

• Select the appropriate label in the Results list and click Close button. If you have a large number of builds, you may need to limit the scope of search by owner and label name.
• Back in the Get dialog, click Get button. Visual Studio will download a snapshot of the code that was compiled by Team Build and released to production.

Make changes required to fix the problem

• Before you open your solution, make sure that you don’t have “Get everything when a solution or project is opened” setting turned on.

Having downloaded the snapshot of production source code in your workspace, you can now modify it to fix the production problem. You will need to check in the changes you made in order to make them available for Team Build. Note that any changes you check in apply to both current and released version.

In addition to fixing the problem itself, you will also need to modify the Team Build definition. Normally, Team Build will automatically generate a new build number based on the current date. You will need to modify the Team Build definition file to allow overriding build number using MSBuild command-line parameters.

• Add the following XML to the build definition file. Build definition files are typically called TFSBuild.proj and located in the $/<ProjectName>/TeamBuildTypes/<BuildName> directory of your project in source control.

<!– Define property to override default build number  –>
<PropertyGroup>
    <OverrideBuildNumber></OverrideBuildNumber>
</PropertyGroup>

<!– Override automatically generated build number –>
<Target Name="BuildNumberOverrideTarget"
        Condition="‘$(OverrideBuildNumber)’ != ”">
    <CreateProperty Value="$(OverrideBuildNumber)">
        <Output TaskParameter="Value" PropertyName="BuildNumber" />
    </CreateProperty>
</Target>

Note this fragment defines a custom MSBuild property, OverrideBuildNumber, we will later use to specify the specific hot fix version of source code Team Build needs to compile. It also overrides a standard BuildNumberOverrideTarget defined by Team Build to override the automatically generated, standard BuildNumber property when OverrideBuildNumber is specified.

Although exact placement of the PropertyGroup and Target elements within the parent Project element inside of TFSBuild.proj shouldn’t matter, it is a good idea to place the OverrideBuildNumber property close to other properties in the beginning of the file, and the Target element close to other tasks in the end of the file. MSBuild files follow the traditional interface/implementation source file layout.

Apply label to identify the hot fix version of source code

Normally, Team Build will automatically get the latest version of source code for each build and label it once the build is completed. However, to build a snapshot of the production source code with the hot fix applied, we will need to label it in our workspace first. Team Build will then use this label to download it from source control.

• Right-click the project root in the Solution Explorer and select Apply Label from the context menu
• In the Choose Item Version dialog, select “Workspace Version” and click OK.

• In the Apply Label dialog, enter name of the hot fix label and click OK.

Name of the hot fix label will be typically based on the label of the last production build. In this example, the last production build was labeled as FirstBuild_20090414.1, so we use FirstBuild_20090414.2 for the hot-fix build. Although in your case, the numbering scheme may be different, the general idea of incrementing the revision portion of the build number should still apply.

Make Team Build get and build the hot fix version

• Right-click the build in Team Explorer and select Queue New Build item from the context menu.
• In the Queue Build dialog, use the “MSBuild command-line arguments” text box to specify values for GetVersion and OverrideBuildNumber properties.

GetVersion is a standard property that specifies which version Team Build needs to pull from source control for the build. It’s value matches format of the VersionSpec parameter of the tf get command. In this example, it needs to be LLabelName, where (first) L is a required prefix and LabelName is the name of the hot fix label we applied previously. OverrideBuildNumber is the custom property we defined in our TFSBuild.proj.

• Click the Queue button.

Unless there were some error made in TFSBuild.proj or the source code, the build should complete successfully.

Results

The sample source code accompanying this article illustrates the Workspace approach. It reflects the following scenario.

• On April 14, development team delivers a build labeled FirstBuild_20090414.1 to QA team for testing.
• On April 15, development team begins actively working on features for the next major version.
• On April 28, after a 2 week testing cycle QA team releases build 20090414.1 to production.
• On May 14, a developer checked in implementation of a new feature, scheduled for the next major release.
• On May 14, development team delivers a build labeled FirstBuild_20090515.1 of the new major version to QA team for testing.
• On May  21, a critical production bug was identified.
• On May 21, development team fixes the production problem and delivers a hot fix build labeled FirstBuild_20090414.2 to the QA team for testing.
• On May 22, After a 4  hour testing cycle QA team releases hot fix 20090414.2 to production.
• On May 22, development team resumes working on the new version and delivers new build labeled FirstBuild_20090522.2 to the QA team for testing.

Change History

Note the changeset numbers shown below.

Changeset 220 contains changes for the next major version. In particular, it contains Form1.Designer.cs file, where a new Label was added. Change 221 contains changes required for the hot fix. It contains a modified version of Form1.cs.

Build History

Note that the hot fix build 20090414.2 was done after a new feature changes was checked in on 5/14 in changeset 220.

Below you can see what got built in the hot fix build 20090414.2. Although Form1.Designer.cs was modified on 5/14 to implement new features, note that this build contains the original version of this file (changeset 218) and new version of Form1.cs (changeset 221), which was modified for the hot fix.

And here is what got built in the next regular build 20090522.1 of the new version on May 22. Note that it contains new versions of both Form1.Designer.cs (changeset 220) modified earlier and Form1.cs (changeset 221) modified for the hot fix. In other words, changes made for the hot fix were immediately applied to the main codebase of the next major version of the application.

Finally, below are the contents of the drop directory. As you can see the drop directory for build 20090414.2 was successfully created on 5/22 when the build was performed.

Conclusion

Approach described in this article allows to create a hot-fix for a Team Foundation build created in the past without introducing untested side effects and without branching the source code. This approach relies on a TFS workspace to provide temporary isolation of the source code required to apply the hot fix. Because this isolation is temporary, the workspace approach will not work for more complex scenarios, such as when a file where hot fix needs to be made is already modified in the new version or has been renamed. On the other hand, in simple scenarios, where ongoing parallel maintenance of the production version is not necessary, this approach offers a simpler solution that doesn’t impose added complexity of branching and merging on the development team.

The workspace approach can also be combined with the branching approach. The workspace approach could be used to release the initial hot-fix build. The development team can then switch to the branching approach when necessary. In this case, development team would branch the source code based on the label of the last hot-fix build.

Coincidentally, having the OverrideBuildNumber property in your Team Build script also gives you ability to recreate any previous build, not just a special hot-fix build. Imagine going through a long test cycle just to find out that a build you were about to deploy to production was accidentally deleted. Of course, having this ability in your Team Build script is not a substitute for a proper backup procedure. You may not be able to recreate the exact same build output if the Team Build script itself or the Build Server has changed since then.

Download

• Source code