Welcome to EMC Consulting Blogs

Anyone who has done anything moderately complex with .NET will have undoubtedly felt the pain associated with having to deal with your application requiring different configuration settings when deployed into to different environments – and if you’re working with WCF then even more so! (Clearly this is a general pain felt across all flavours of development and whilst for purposes of this post I’m focussing on the .NET world, the core approach could be applied more broadly)

Here at EMC Consulting this challenge has typically fallen to us Platform Architects to overcome, and over years we’ve evolved some in-house tooling that could integrate with our build and deployment processes whilst at the same time trying to minimise any additional overhead on the day-to-day development process.  The various evolutions all followed this basic premise:

• Configuration settings that need to be environment-specific are replaced with tokens
• Maintain a set of mappings that assign the required values for each token, for a given environment
• During deployment, use a simple search & replacement operation to patch any tokenised configuration files with the required values

The need for developers to have valid configuration files to support local their local build & debugging activities, meant that earlier iterations involved maintaining a template copy of each configuration file that contained environment sensitive settings.  Whilst this gave us a good story for not impacting the day-to-day developer activities, it did bring the additional overhead of having to keep the two files in-sync.  Inevitably, this would not always happen (particularly when the configuration change was not of an environment-specific nature) and thus the template version would become out of date and ultimately result in a deployment failure later on.

It was this aspect that led me to think about alternative approaches that could reduce this overhead. Several years ago I remember seeing a cool approach for doing this that formed part of the Enterprise Library (EntLib) – whilst still relying on a separate configuration file it used a mechanism that permitted target configuration files to be transformed based on criteria in this separate configuration file (which only had to contain the elements that needed to change, rather than a complete copy of the target configuration file).

Specifically, this approach expected you to define a transform for each environment where you would apply the required settings, but I figured you could equally create a single transform and use the tokens mentioned above instead and then run the search & replace operation against the transformed configuration file.  The main downside of this tool, in my view, was that it seemed to only work for custom configuration settings implemented via EntLib so could not be used to manage standard web.config settings, for example.

I would hazard a guess that the above EntLib functionality was the basis/inspiration for the similar functionality we now see in the MSDeploy ‘configuration transform’ feature.  However, in terms of the out-of-the-box experience this feature is limited to web.config files, although there are various posts out there that show how to use it more broadly (http://www.olegsych.com/2010/12/config-file-transformation/).  Again, this approach could be combined with the use of tokens to avoid maintaining multiple transform files.

With that background out of the way let’s get to the title point of this post – using T4 templates to address the issues outlined above.

If you’d like some more information of T4, then here are a few links to get you started:

• http://msdn.microsoft.com/en-us/library/bb126445.aspx
• http://www.hanselman.com/blog/T4TextTemplateTransformationToolkitCodeGenerationBestKeptVisualStudioSecret.aspx
• http://www.olegsych.com/2007/12/text-template-transformation-toolkit/

The essence of this approach is just an extension of the one above, and it looks like this:

• Convert all configuration files that need to be environment-specific to be T4 templates
• Implement T4 logic that can inject into these configuration files, either:
  1. actual values (to support local build & debugging activities)
  2. tokens (for deployment packages)
• Utilise the T4 support built into Visual Studio to regenerate the config file using the #1 scenario above, whenever the file is modified
• Have the build process execute the T4 logic using the #2 scenario to create the tokenised configuration files
• Maintain a set of mappings that assign the required values for each token, for a given environment
• During deployment, use a simple search & replacement operation to patch any tokenised configuration files with the required values

This allows a single copy of that configuration file to be maintained (albeit in a modified form) and also has the added benefit of placing the environment-specific nature of the configuration file in clear view of anyone working on that file, hopefully making it more likely that they will think about the broader configuration concerns rather than just whatever it takes to get it working for them.

In closing, at the risk of being hit with the ‘Not Invented Here’ stick, whilst the latest tools from Microsoft (config file transformations) are a welcome addition to the stock toolbox, they are limited to being used with XML files (and MSBuild seemingly) – which whilst representing the majority of scenarios does leave some edge cases unsupported.  As with most things in our industry there are many ways to solve this problem, and each has their own pros and cons and I present this as just another option.

In my next post I’ll run through a sample implementation of the above.

In the meantime, if you have any comments on this approach or can suggest other alternatives that have worked for you then please drop me a comment.

Cheers, James.
(@James_Dawson)

A few weeks ago James sent me an e-mail saying that he’d been coming across this thing called ‘DevOps’ a lot lately and it seemed like it might be the closest thing yet to our Platform Architect role, outside of EMC Consulting.

The Platform Architect team is a small, and mostly merry, band of technical consultants who typically started out their careers on the Infrastructure side of IT and have evolved to straddle the fence that divides the Infrastructure and Development worlds - if you read Grace’s blogs, then you’ll already have a fair idea of the types of things we get up to.

So getting back to my introduction to the DevOps world, I was intrigued by it not least because I often struggle to define what we as PA’s do when talking to Clients without getting bogged-down in details; such is the breadth of what we have to cover.  Hence why coming across a term with a defined understanding in the outside world was of such interest to me, both in terms of talking to Clients as well as how we find and recruit new PA’s.

Hunting around for some further reading, one of the first articles I came across, was a guest post by Stephen Nelson-Smith on the rather coolly-named jedi.be blog (Just Enough Developed Infrastructure – nice!) which resonated with me hugely.  As consultants, clearly we’re not sysadmins in the purest sense, but we are invariably the sysadmins for those projects that we’re involved with and as such the same pain points and desire to make things better still ring true.  However, one of the benefits of operating within the team at EMC Consulting is that the typical barriers between disciplines that Stephen mentions are far less prevalent, as is the siloisation, mainly due to working in small to medium-sized project teams.  That said, we will often collaborate with the Client’s various teams and it is here that we regularly see exactly the tensions between them that Stephen highlights, and in that respect our role is very much a bridging one too.

One thing that seems notably absent from what I’ve read about DevOps (to-date) is any mention of Application Lifecycle Management (ALM) – this forms a fairly large part of the PA role here and I’m keen to learn whether ALM is indeed considered outside of DevOps, or whether I just haven’t been looking in the right places.  On the surface, much of what is talked about seems relevant to different parts of ALM but no-one seems to be making any explicit references to it.  I’d be really interested to hear from anyone who can shed some light on this, one way or the other.

Configuration Management seems, understandably, to be a key topic in the DevOps world so in my next post I want to discuss an approach for tackling application configuration management that I’ve been working on lately.  I use a lowercase C & M on purpose, as it is very much a lightweight approach that aims to remove as much friction as possible from the task of making application configuration files environment-agnostic (e.g. not hard-coding server names, credentials, paths etc.) whilst at the same time ensuring that the need for this activity is still very much front-and-centre in the development team’s mind.

Cheers,

James.

(@James_Dawson) 

So, there’s been a bit of storm blowing through the part of the InterWeb that I inhabit recently – see http://blog.benday.com/archive/2009/07/27/23233.aspx for the low-down.

I’ve seen these types of discussions countless times, both online and in the real world – the thing that wears me down the most is the seemingly constant assertion that ‘SVN/Git/Mercurial = TFS’, as if source control is the only consideration.

Let me lay down a couple points of order before I get flamed to oblivion:

1. I’ve generalised this assertion, somewhat
2. I’ve spent time using both SVN and Git and like them both
3. I have no desire to sound like a Microsoft shill!

Fundamentally, SVN and TFS are implementations of a centralised version control system (VCS) that are designed using polar-opposite philosophies – TFS is server-centric whilst SVN is client-centric.  The reality is that both of these approaches have advantages and disadvantages in terms of usability and performance.

Regarding the centralised versus distributed debate, I actually like a lot of things about Git et. al. and I would guess that as the various tooling/integration stories develop we will see more corporate and enterprise adoption.

Getting to my point though, most of the customers I talk to rarely arrive at the TFS ‘answer’ on the basis of a VCS selection process (those that have, usually get a suggestion from me to look at other open source alternatives), they get there because they are looking for something to satisfy their Application Lifecycle Management (ALM) needs – I consider TFS to be pretty cost-effective in the ALM space.  Granted, there is a sizable chunk of TFS adopters who migrate from Visual SourceSafe (and are pre-disposed to use Microsoft tools) but as a like-for-like swap Microsoft have, in general terms, made that migration relatively painless.

However, increasingly we’re seeing customers who have arrived at the TFS ‘answer’ because they use Scrum to manage their development projects and have chosen Scrum for Team System (a TFS process template we make available for free) as their preferred project management tool - and from their perspective TFS comes with the added bonus of not having to worry about selecting and integrating a separate VCS and automated build system.

A customer recently contacted me saying that they had upgraded to Visual Studio 2008 and had also taken the opportunity to switch from a Web Site Project to a Web Application Project.  A while back I'd helped them to centralise some of their configuration so that they only had a single copy of all the configuration sections that were referenced by multiple projects (e.g. the web site, various unit test projects etc.).

This involved creating separate configuration files for each section and referencing them from the main web/app config using the 'configSource' attribute, for example:

The above referenced configuration file would something like this:

Due to the 'configSource' attribute not supporting relative paths, it was necessary to setup pre-Build events to copy the referenced files into the project folder - especially for the Web Site project where a valid web.config is required by the ASPNET compiler.

After the switch to the Web Application project model the client was wondering whether they needed to change how they were handling these shared configuration files.  My initial response was to suggest that they discontinue the pre-build scripts and instead use the 'linked files' feature that was available with web application projects (much like they have been for other types of projects a long time).

For those of you unfamiliar with this useful feature, here's how you add a linked file via the 'Add Existing Item' option:

Such files appear in Solution Explorer as normal project items, albeit with a shortcut icon.

However, after trying it, they reported that those files were missing from the web site which was causing all manner of problems.

It turns out that when you build the project linked files are not copied into the project folder, which is obviously a problem as it means that the file is not available to the web site at runtime.  Your first thought might be to enable the 'Copy Local' option for the linked files, however, all this does is to copy the linked files to the 'bin' folder... not exactly the result we're after.

Before carrying on, it's worth looking at how the linked items are stored in the actual web application project file, as it will later help us understand why this issue occurs.

From this you can see the main path for the item (the 'Include' attribute) is a relative path to the shared configuration file, whilst the 'Link' attribute tells Visual Studio where the item exists within the project - in this case, the file was added to the root of the project.

To investigate further I took a look at the 'Microsoft.WebApplication.targets' file that contains the Web Application project build process and typically lives in the 'C:\Program Files\MSBuild\Microsoft\VisualStudio\v9.0\WebApplications' folder.  There I found a target called '_CopyWebApplication' which only gets executed when the output directory has been overridden (e.g. what TeamBuild does to redirect all build outputs into a single directory):

At the end of this Target there is the following call to the MSBuild Copy task:

So when the above Copy task executes it's going to use the relative path to the linked item as part of the destination path, rather than the path within the project (i.e. the 'Link' attribute) - so this explains why the files aren't getting copied:

1. When building in Visual Studio, nothing special happens to copy the content files (linked or otherwise)
2. When redirecting the output path, normal Content files are copied fine (their 'Include' attribute does not contain a relative path), but the linked content files will get copied to a folder with the same relative path to the actual output folder, as the linked item has to the project file (e.g. 'MyOutDir\..\..\dbConfig.config' instead of 'MyOutDir\dbConfig.config')

Anyway, let's cut to the chase... here is my suggested workaround:

1. Fix the 'Copy' task to not include linked files
2. Create another target to handle the copying of linked files using the correct destination path

To achieve the first, we'll need to override the built-in '_CopyWebApplication' target by pasting it into our web application's project file and tweaking the 'Copy' task mentioned above:

  <!--

  ============================================================

  _CopyWebApplication

  MODIFIED: Ignores linked files as part of normal deployment logic.

  This target will copy the build outputs along with the

  content files into a _PublishedWebsites folder.

  This Task is only necessary when $(OutDir) has been redirected

  to a folder other than ~\bin such as is the case with Team Build.

  ============================================================

  -->

  <Target Name="_CopyWebApplication" Condition="'$(OutDir)' != '$(OutputPath)'">

    <!-- Log tasks -->

    <Message Text="Copying Web Application Project Files for $(MSBuildProjectName)" />

    <!-- Create the _PublishedWebsites\app\bin folder -->

    <MakeDir Directories="$(WebProjectOutputDir)\bin" />

    <!-- Copy build outputs to _PublishedWebsites\app\bin folder -->

    <Copy SourceFiles="@(IntermediateAssembly)"

          DestinationFolder="$(WebProjectOutputDir)\bin"

          SkipUnchangedFiles="true" />

    <Copy SourceFiles="@(AddModules)"

          DestinationFolder="$(WebProjectOutputDir)\bin"

          SkipUnchangedFiles="true" />

    <Copy SourceFiles="$(IntermediateOutputPath)$(_SGenDllName)"

          DestinationFolder="$(WebProjectOutputDir)\%(Content.SubFolder)%(Content.RecursiveDir)"

          SkipUnchangedFiles="true"

          Condition="'$(_SGenDllCreated)'=='true'" />

    <Copy SourceFiles="$(IntermediateOutputPath)$(TargetName).pdb"

          DestinationFolder="$(WebProjectOutputDir)\bin"

          SkipUnchangedFiles="true"

          Condition="'$(_DebugSymbolsProduced)'=='true'" />

    <Copy SourceFiles="@(DocFileItem)"

          DestinationFolder="$(WebProjectOutputDir)\bin"

          SkipUnchangedFiles="true"

          Condition="'$(_DocumentationFileProduced)'=='true'" />

    <Copy SourceFiles="@(IntermediateSatelliteAssembliesWithTargetPath)"

          DestinationFiles="@(IntermediateSatelliteAssembliesWithTargetPath->'$(WebProjectOutputDir)\bin\%(Culture)\$(TargetName).resources.dll')"

          SkipUnchangedFiles="true" />

    <Copy SourceFiles="@(ReferenceComWrappersToCopyLocal); @(ResolvedIsolatedComModules); @(_DeploymentLooseManifestFile); @(NativeReferenceFile)"

          DestinationFolder="$(WebProjectOutputDir)\bin"

          SkipUnchangedFiles="true" />

    <!-- copy any referenced assemblies to _PublishedWebsites\app\bin folder -->

    <Copy SourceFiles="@(ReferenceCopyLocalPaths)"

          DestinationFolder="$(WebProjectOutputDir)\bin"

          SkipUnchangedFiles="true" />

    <!-- MODIFICATION HERE: Copy local content files (i.e. non-linked files) recursively to _PublishedWebsites\app\ folder -->

    <Copy Condition=" '%(Content.Link)' == '' "

          SourceFiles="%(Content.Identity)"

          DestinationFolder="$(WebProjectOutputDir)\%(Content.RelativeDir)" />

  </Target>

For the second, we need to add a new target to our project file and override the default dependencies to have it executed:

  <!--

  ============================================================

  CopyLinkedContentFiles

  A new target to copy any linked content files into the

  web application output folder.

  NOTE: This is necessary even when '$(OutDir)' has not been redirected.

  ============================================================

  -->

  <Target Name="CopyLinkedContentFiles">

    <!-- Remove any old copies of the files -->

    <Delete Condition=" '%(Content.Link)' != '' AND Exists('$(WebProjectOutputDir)\%(Content.Link)') "

            Files="$(WebProjectOutputDir)\%(Content.Link)" />

    <!-- Copy linked content files recursively to the project folder -->

    <Copy Condition=" '%(Content.Link)' != '' "

          SourceFiles="%(Content.Identity)"

          DestinationFiles="$(WebProjectOutputDir)\%(Content.Link)" />

  </Target>

  <!-- Override the default target dependencies to -->

  <!-- include the new _CopyLinkedContentFiles target. -->

  <PropertyGroup>

    <PrepareForRunDependsOn>

      $(PrepareForRunDependsOn);

      _CopyWebApplication;

      CopyLinkedContentFiles;

      _BuiltWebOutputGroupOutput

    </PrepareForRunDependsOn>

  </PropertyGroup>

With these changes, whenever you build the web project any linked content files will be copied into the web application's folder structure  - whether you override the default output directory or not.

If anyone is interested I've attached a sample solution that demonstrates the above workaround (as well as an unmodified project file so you can see the issue).

Part of me wonders whether this is a bug at all, or whether it is in fact just an unexpected feature that was never supposed to be supported... I've raised a bug on Connect, so I guess I'll find out in due course.

This post started out as a comment on one of Jamie's recent posts, however, I realised that the explanation would be easier to follow with screenshots... so here we go.

To paraphrase, Jamie's issue was that whilst trying to use an SSIS package he was getting a DCOM access denied error (full details here) - this had worked in previous environments, but not once it got to Production.  The eventual solution that got everything working was to make the account running the package a local administrator (as it was in other environments) and reboot the server.

Whilst this obviously got things working, what I'd like to illustrate here is a more secure way of overcoming this type of issue - as a platform guy, developer code running with full admin rights makes me twitchy!

DCOM security is managed using the 'Component Services' MMC snap-in, a shortcut to which is 'Start -> Run -> DCOMCNFG'

The next step is locate the component that needs to have its security tweaked.  Jamie's post shows how to use the CLSID from the error message to locate the registration information for the component in the Registry.

The value of the ''AppID' key is the surefire way of locating the component in DCOMCNFG, however, this is only visible from the component's property page - so ideally you want to find
the component by name.  However, the names in DCOMCNFG do not necessarily exactly match those in the registry (typically it's a substring).

Armed with the AppID and some ideas about the component name, return to DCOMCNFG and start hunting:

Once you've found a candiate, open its properties and verify the AppID

If you've found it, select the 'Security' tab, select 'Customize' and 'Edit..' under 'Launch and Activation Permissions', and grant the required user the rights requested in the original error message.

Now you should be good to go. (complete with that warm and fuzzy feeling that you get from knowing that you haven't unduly compromised server security)