ALM Adventures

In a perfect world, we would all be working on high quality greenfield projects using all the latest technologies and development practices. But often (in my experience at least) we are called on to support applications that have grown uncontrollably for many years, carrying enough technical debt to make Greece blush.

But even these monstrosities can benefit from modern ALM practices such as automated build and deployment. So in this post I’m going to talk about automated deployment of a real-life legacy web application using scripting, Psexec and other command-line tools – all coordinated by TFS 2010.

The Scenario

The application had been developed chaotically by many people over a decade or more using every Microsoft technology you could name: C++, VB, VBScript, VBA, C#, .NET 1.1, .NET 2.0, ASP, ASP.NET… you name it, it was in there somewhere. There had never been a rewrite, re-architecting or even any hints of refactoring. Yet development on this key application continued, and while there was an appreciation that things needed to improve there was certainly no appetite for any major rework.

The Challenge

The goal was to improve quality and productivity, and reduce risk. I know it’s a cliché, but the way to eat an elephant is one bite at a time, and that is how I approached this problem. The steps below were taken to tame this beast, tentatively, over several months while development continued relatively uninterrupted:

1. Migrated source control from Visual SourceSafe to TFS 2010 (there is plenty of documentation on this, just make sure you practice before doing the real migration).
2. Productised the application – defined the elements that it is (and isn’t) made of, and got them all organised in source control to establish a “single source of truth”.
3. Migrated issue tracking from a proprietary system to TFS 2010 Work Items so changes going forward are transparently tracked.
4. Got the application to compile with a minimal (but fully documented) set of components required to be installed. Later work cleaned up referenced dependencies, which were scattered and sometimes in conflict.
5. Introduced TFS automated builds, starting with continuous integration.

So far, so good. By this point, we knew what we were dealing with and were building it on every check-in. There were even unit tests – which were patchy in coverage and quality, but much better than nothing.

The next step was to get this caged monster deploying to a test environment, fully automated, starting from a standard base configuration. The existing  process was documented as manual steps over dozens of pages, and it still wouldn’t work first time – days would be lost “setting it up” after following the install procedure. If you can’t easily deploy your application onto fresh hardware, then something is seriously wrong.

The benefits of a clean and endlessly repeatable deployment process are many, but getting there wouldn’t be easy.

WebDeploy

An early decision was made to restrict the deployment to a collection of ASP.NET web applications, as this was the more modern part of the app, and under constant development. The latest and greatest technology for this is the unimaginatively named Web Deployment Tool, also known as WebDeploy and MSDeploy, which is built on top of MSBuild – Microsoft’s core compilation technology.

We spent a good deal of time working with WebDeploy, but it turned into a battle. The constraints of old application and OS technology, and multiple web applications were a big headache and the pretty Visual Studio tooling soon fell short, leaving us working directly with MSBuild files.

Personally, I like the change of direction in TFS 2010 to restrict MSBuild to compilation, and use Windows Workflow for build “orchestration”. Once you go beyond compilation and directly related activities, changing XML-based MSBuild scripts feels awkward compared to the intuitive free-flow of WF (the abbreviation is not WWF – I think even Microsoft is scared of the World Wide Fund for Nature). So we ditched WebDeploy and decided to roll our own TFS Build and scripting solution.

The Solution

I came across many issues as I put our deployment solution together, so I will set the whole process out here and pick out the problems.

When to Deploy

The default TFS build process template is necessarily complex, and I intuitively put an Invoke Process activity to start the deployment after the steps which compile and test the solution. However, our solution had functional tests (calling high-level methods which use web services and the database) so you would get test failures because the tests relied on the deployment itself. So I simply moved the deployment step before the tests, and ensured any parts which the tests depended on would run synchronously so they finished before the tests started (more on this later).

There is a potential downside to this, which is that the compile and test steps are within a for-each loop for each solution and configuration combination. If you compile more than one solution and/or configuration in your build definition (which I personally don’t like to do) it will get messy. The deployment would run multiple times, but if you added a condition of some sort to ensure it runs only after the last compilation, it might be too late for any tests which depend on it. I didn’t need to deal with this issue, but controlling the order of compilation and breaking up the deployment seems like a solution. Another option would be to have multiple builds, each with a single solution and configuration, and have one build trigger the next.

Passing Parameters

It is very useful to pass information from the build definition and build system through to the deployment scripts. To achieve this I created a handful of new arguments in a new “Deployment” category, including:

• An on/off toggle
• Email addresses to notify of build completion
• Processes to invoke to do the deployment
• Text about the deployment to display in the build log

The processes argument could include placeholders in curly braces, which the build template was customised to recognise and replace appropriately. For example, placeholders of {BUILDNUMBER} and {BUILDID} would be substituted by the actual build number and numeric build ID respectively. The scripts could use this information to stamp the website with a build number and send an email with a link to the TFS Web Access version of the build log. (Incidentally, it’s an annoyance for a pedant like me that the build number is not a number, that’s the build ID – am I alone in being bothered by that?)

.

Strategy

I settled on a fairly simple process that worked for the first application and all those it was extended to. It used a file share on the target server as a working folder, and went like this:

1. Clear the file share on the target server
2. Copy the scripts and any other files to the file share on the target server
3. Repeat steps 1 and 2 if there are multiple servers in the deployment
4. Stop application websites and start the maintenance website on the remote server(s). I used Psexec for this initially but found it unreliable (more on that later) so switched to iisweb with much better results
5. Package the application into one or more zip files.
1. I used the 7-Zip command line tool 7za.exe, and a combination of copy and xcopy. Temporarily I used the incredibly flexible robocopy to deal with the horrible consequences of referencing different versions of the same assembly from different projects in the same solution – the application was fixed properly later.
2. Any web.config or app.config transforms will need to take place before this step. I posted previously about how to trigger this without a full publish action.
6. Copy the zip package(s) to the target server(s)
7. Run the deployment actions in an inside-out sequence. So in a three-tier architecture: database, then the application layer, then the UI layer.
• I used Psexec to asynchronously hand over the deployment to each target server. This takes the load off the build server and means the deployment runs as a process local to the server, which can be helpful for security and performance reasons.
• If you want the build server to wait for the some or all of the deployment to finish, either run the deployment remotely from the build server or use a signal such as a “semaphore file” to tell the build server when the process is complete (see the ALM Rangers build guidance for more on that).
• If using 7-Zip to extract zip files use the -y switch to avoid prompts hanging the process.
8. When the deployment actions were complete, the final server in the deployment sequence was scripted to:
1. Stop the maintenance website and start the application’s website
2. Copy all the generated logs into the build drop folder
3. Send an email notification of the deployment

Preparatory steps and signposting of deployment steps were output into the build log itself, but process calls with extensive output were set to use log files instead. The build log doesn’t lend itself well to very verbose logging.

Psexec

This is a command line tool from Sysinternals which performs the magic of triggering a process on a remote machine. It is very temperamental about how it is used but not unreliable once you find a stable usage pattern. Here are the lessons I learned:

• Admin shares (\\server\C$ style file shares) need to be enabled on the remote machine.
• This requires some one-time config changes if the server has been hardened against it (which should only be done on an internal network with appropriate approval). I needed to set Server and Computer Browser services to run automatically, set registry key HKLM\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters to 1, and kill/restart the explorer.exe process.
• The full path to Psexec is needed for it to be found by the build agent (even if it is under the PATH environment variable)
• cmd /c is necessary to run Psexec from an InvokeProcess build activity as it is a console command, not a standalone executable
• Psexec returns its own output as StdErr, causing the build log to contain an error for every line of output – and failing the whole build. You can use output handler 2>&1 to push StdErr into StdOut.
• Psexec likes to output empty lines but you can strip them out within (or before) a WriteBuildMessage activity
• For reliability I found I needed switches -d and -accepteula to run non-interactively and accept the embedded license agreement (the latter is common to Sysinternals tools).
• I used Psexec only once per server, which seemed necessary to avoid overlapping processes (I don’t know why that should have been a problem, but it caused poor reliability). Having this constraint actually helped ensure the deployment process was well structured.
• If you don’t specify a user account, Psexec uses impersonation which may result in the remote machine lacking access to network resources. This can be avoided by explicitly passing the username and password.
• To pass the output of a remotely executed command to a log requires a middle-man script on the remote server. If you try to log on the calling server, you will get the output of Psexec itself, not the command being called.

Shared Deployment Scripts

This deployment strategy was later reused for other applications. To avoid duplication of componentised scripts and other shared items, I moved them to their own source folder and included them in the workspace of the build definition. You can’t map two source folders into the same working folder, so you have to figure out the correct relative paths, but it’s a small price to pay for the elimination of duplication (it’s evil!)

Summary

I will readily admit that this deployment approach is a little archaic. It’s also littered with gotchas, but new technology is not immune from that either.

Going forward, WebDeploy is still my first choice whenever possible. But if faced again with a trade-off between extensive MSBuild scripting and relatively transparent TFS Build + Windows scripting, I would still go for the latter.