Database deployments with Octopus and Redgate SQL Release

If you work with SQL Server, you're probably familiar with our friends at Red Gate. They're the leading developer of SQL server tooling. While some of their best known tools (like SQL Compare) focussed on making manual processes easier, lately they've invested a lot of time and energy into database automation. Their latest creation is a plugin called SQL Release, which integrates with Octopus to enable automated database deployments. The following is a guest post by Brian Harris from the SQL Release team.

What is SQL Release?

On the SQL Release team we want to make releasing a new version of your database as easy as releasing other applications. The trouble is that databases contain data; so rather than deploying an entirely new database, you need to upgrade the state of the live database state to match the new version.

The solution we’ve implemented with SQL Release is to generate a SQL update script that upgrades the existing database to the new version. Before running this script, you can review the changes that will be made, and look at any warnings that are raised (for example, data loss when dropping a table).

When you’re happy with the update, SQL Release can run the update script. For additional security, before running the script, SQL Release checks that the target database has not changed since the script was generated.

SQL Release is implemented as a series of PowerShell cmdlets for easy integration with Octopus Deploy. In the future we plan to integrate with other release management tools.

Walkthrough: Using SQL Release with Octopus Deploy

To illustrate how this works in practice, we’ll go through a simple scenario using SQL Release and Octopus to deploy directly from a development database to a production database.

Before we start, we need the following to be set up:

  • Octopus Deploy
  • An Octopus Deploy environment called Production
  • An Octopus tentacle configured with the role of db-server
  • SQL Release installed on the same machine as the Octopus Tentacle (the Tentacle needs to be restarted after installing SQL Release)

Creating a new Octopus Deploy project

First we need to create a new Octopus project and give it a name. The project consists of a series of steps which manage the process of deploying the database. When we’re done it will look like this:

Redgate SQL Release Octopus Integration

Note that we don’t have to specify a target environment for any of the steps in this project. The project will deploy to all environments by default if you do not specify an environment, which in this example is fine, as there is only a single Production environment.

Adding the "Make database update resources" step

The first step we add creates the deployment SQL script. On the project Process tab, select Add step and select Run a PowerShell script.

Enter these details in the fields:

Name: Make database update resources
Machine roles: db-server
Script: This script uses a set of project variables that we will define later:

# This step uses SQL Release to create a directory containing the
# Database Update Resources: all the artifacts relating to the deployment.
#
# The directory, 'DatabaseUpdateResources', has the structure:
#   - States
#       - BeforeUpdate.snp: SQL Compare snapshot of the state the database
#                            is in BEFORE the deployment.
#       - AfterUpdate.snp: SQL Compare snapshot of the state the database
#                           will be in AFTER the deployment.
#   - Update.sql: The SQL change script that updates the target database from the
#                  BeforeUpdate.snp state to AfterUpdate.snp state.
#   - Reports
#       - Changes.html: An HTML report showing which database objects
#                       will be changed in the deployment and how.
#       - Warnings.xml: An XML report containing warnings relating to the
#                        deployment.
# Makes sure the directory we're about to create doesn't already exist.
If (Test-Path $DatabaseUpdateResourcesDirectory) {
    rmdir $DatabaseUpdateResourcesDirectory -Recurse -Force
}
# Sets up connection string for the target database.
$developmentDatabase = "Data Source=$DevelopmentDatabaseServer; `
                        Initial Catalog=$DevelopmentDatabaseName; `
                        User ID=$DevelopmentSQLServerUsername;Password=$DevelopmentSQLServerPassword"
$productionDatabase =  "Data Source=$ProductionDatabaseServer; `
                        Initial Catalog=$ProductionDatabaseName; `
                        User ID=$ProductionSQLServerUsername;Password=$ProductionSQLServerPassword"
# Creates the DatabaseUpdateResources directory.
New-DatabaseUpdate -BeforeUpdate $productionDatabase `
                   -AfterUpdate $developmentDatabase `
                   -Verbose `
| Export-DatabaseUpdate -Path $DatabaseUpdateResourcesDirectory
# Imports the changes report, deployment warnings, and update script
# as Octopus artifacts, so you can review them in Octopus.
New-OctopusArtifact "$DatabaseUpdateResourcesDirectory\Reports\Changes.html"
New-OctopusArtifact "$DatabaseUpdateResourcesDirectory\Reports\Warnings.xml"
New-OctopusArtifact "$DatabaseUpdateResourcesDirectory\Update.sql"

Click Save.

Adding the "Review database update resources" step

Next we add a step to pause the deployment for manual review of the script and other resources, so we can make sure we’re happy with the release before it goes ahead. The previous step imported a Change Report, the Update Script, and a Warnings file, which will be visible in the Octopus UI for review.

On the project Process tab, click Add step and select Manual intervention required.

Enter these details in the fields:

Name:
"Review database update resources"

Instructions:

Please review the deployment artifacts on the right:
(1) Update.sql: SQL change script that updates the target database.
(2) Warnings.xml: XML report containing warnings relating to the deployment.
(3) Changes.html: HTML report showing which database objects will be changed in the deployment and how.

Adding the "Run the update script" step

The final step deploys the script we’ve generated upon approval of the review step.

On the project Process tab, click Add step and select Run a PowerShell script. This is the deployment step.

Enter these details in the fields:

Name: Run the update script
Machine roles: db-server
Script:

#This step uses SQL Release to deploy the database update we previously generated and wrote to disk.
$targetDatabase = "Data Source=$ProductionDatabaseServer; `
                   Initial Catalog=$ProductionDatabaseName; `
                   User ID=$ProductionSQLServerUsername;Password=$ProductionSQLServerPassword"

Import-DatabaseUpdate $DatabaseUpdateResourcesDirectory | Publish-DatabaseUpdate -Target $targetDatabase

Click Save.

Set up the project variables

Variable name Value
BaseDirectory #{Octopus.Tentacle.Agent.ApplicationDirectoryPath}\#{Octopus.Environment.Name}\#{Octopus.Project.Name}\#{Octopus.Release.Number}
DatabaseName <NameOfYourDatabase>
DatabaseServer <YourSQLServerInstance>
DatabaseUpdateResourcesDirectory #{BaseDirectory}\DatabaseUpdateResources
Octopus.Action.Package.CustomInstallationDirectory #{PackageExtractDirectory}
PackageExtractDirectory #{BaseDirectory}\DatabasePackage 
SQLServerPassword <YourSQLServerPassword>
SQLServerUsername <YourSQLServerUsername>

Running the deployment

We’re now ready to deploy a database update in Octopus. Deploying a database update is the same process as deploying a normal update using Octopus Deploy.

Summary

I’ve shown you a simple use case: deploying from one database schema to another as part of your Octopus Deploy release, but SQL Release can be used in more complex situations.

We can use SQL Release to release to a pre-production environment first, and after validating the deployment was successful, we can promote it to our production environment.

The benefit of this approach is that we can run exactly the same script against production that we tested on our pre-production database. As part of the deployment SQL Release checks the production database is still in the initial state we expect, so we can be sure the trial run against pre-production was a good test of the production release, and the script can be re-used safely.

You can also deploy changes from a database NuGet package. This means it’s possible to source control your database, build and test it in your build server, and then deploy that build using SQL Release, with all the security and reliability that provides.

If you’d like to try SQL Release, you can join our public beta.

Actors vs. RPC: building a new (old) transport layer for Octopus 3.0

In Octopus 1.0, we used WCF to communicate between Octopus servers and Tentacles. When the Octopus server needed to tell Tentacle to do something, such as run a PowerShell script, it looked something like this:

var client = clientBroker.Create<IJobService>("http://some-machine");
var ticket = client.RunScript("Write-Host 'Hello'; Start-Sleep -s 100; Write-Host 'Bye'");

do
{
    var status = client.GetJobStatus(ticket);
    log.Append(status.Log);
    Thread.Sleep(4000);

    if (IsCancelled()) 
    {
        client.Cancel(ticket);
    }
} while (status.State == JobStatus.InProgress);

log.Append("Script run complete");

This RPC style of programming worked well, but it had one limitation: it meant that the Tentacle always had to be the TCP listener, and the Octopus always had to be the TCP client.

When designing Octopus 2.0, the biggest feature request at the time was the ability to have polling Tentacles; effectively, reversing the TCP client/server relationship.

Polling and Listening Tentacles

In the example above, this means that the Octopus server would need to somehow queue a command for Tentacle to run the script, and when Tentacle polls the Octopus for jobs to do, it would find this command and process it. Conceptually, it would mean something like this:

var runScriptCommand = new RunScriptCommand("Write-Host....");
messageQueue.For("MachineA").Enqueue(runScriptCommand);

Using messages decoupled the TCP client/server relationship from the code - we'd be able to write code in Octopus to orchestrate Tentacles, without a whole lot of if/else conditions making it work differently depending whether the Tentacles were listening or polling. Since polling Tentacles required us to have some way of queuing messages to be picked up later, we may as well use them for listening Tentacles too.

There's a big body of knowledge around building distributed systems based on messages and queues, so as soon as we decided that we'd needed queues of messages, those patterns and practices became central to our thinking. Most of our experience with messaging came from frameworks like NServiceBus and similar, which we'd put into practice many times.

Using messages as intended did make the orchestration code more complicated, and our orchestration code started to resemble NServiceBus sagas:

void Handle(BeginScriptRun message)
{
    Send("MachineA", new RunScriptCommand("Write-Host ...."));
}

void Handle(JobStartedMessage message)
{
    State.Ticket = message.Ticket;
}

void Handle(CancelMessage message) 
{
    Send("MachineA", new CancelCommand(State.Ticket));
}

void Handle(ScriptOutputLogged message) 
{
    log.Append(message.Log);
}

void Handle(ScriptCompletedMessage message) 
{
    log.Append("Script run complete");
    Complete();
}

Breaking away from the request/response RPC paradigm to messaging appeared to bring a number of benefits:

  1. It can better handle really long-running tasks, since you don't have a thread blocked waiting for a response
  2. Server uptime is decoupled - if the Tentacle is offline initially, but eventually comes back online, the script run in this example can complete
  3. It allowed us to support both polling and listening Tentacles, since our application code could be written agnostic of the transport underneath

Over time, our NServiceBus saga-like classes evolved into an Actor framework, similar to Akka (though this was about six months before Akka.NET began). We borrowed a number of concepts from Akka, like supervision trees, which made error handling a little more bearable.

This has been in production now for the last 12 months. And while it mostly works well, I've started to notice many downsides to this actor/message-oriented approach:

  1. The single-threaded nature of actors is great and makes concurrency easy. However, there are times where you do want some kind of mutual exclusion, and you end up having to implement it using some ugly messaging approaches anyway.
  2. It's much harder to follow the code and reason about it. Finding which actor handles which message is always a two-step navigation process in Visual Studio.
  3. Crash dumps and stack traces become almost useless.
  4. Managing actor lifetimes is really hard. That Complete() call at the end of the example above is important because it tells us when this actor can be cleaned up. It's so easy to forget to call these (should I have called it in the cancel message handler too?)
  5. Error handling is equally nasty. E.g., in the first code example, if the remote machine failed, the exception would bubble up, or be handled with a try/catch. In the second example, catching and handling these errors is an explicit step.
  6. It's much harder to teach this style of programming to a new developer

When I look at the orchestration code in Octopus 1.6, it's really easy to reason about and follow. Perhaps it doesn't scale as nicely, and perhaps there are a few too many places where we explicitly deal with threading or locking. But the stack traces are readable, and I can navigate it easily. When reading the deployment orchestration code in 2.0, I have to really concentrate.

It also turns out that the "server uptime is decoupled" benefit that I mentioned wasn't terribly useful for us either. If a Tentacle is offline, we might want some automatic retries for, say, 30 seconds or a few minutes. Beyond that, we really don't need messages to queue up. If we're in the middle of a deployment, and a machine is offline, we need to make a decision: either skip it or fail the deployment. We're not going to wait 6 hours. That's usually handled in messaging frameworks by giving messages a very short time-to-live, but it still complicates the exception handling process.

The key benefit of actors is that you gain great concurrency with much simpler code than using the threading/locking primitives directly. And yet when I look at the Octopus 1.0 orchestration code, we use so few of those primitives that the actor approach turns out to be less clean. This suggests that our problem domain isn't really suited for actors. And while we might not be able to escape having messages at some level to handle the polling/listening configuration, request/response semantics would seem to be more suitable.

So, what are we doing for Octopus 3.0? We'll be reverting to something very similar to the Octopus 1.0 style. The difference is that, while from a coding point of view we'll use request/response, underneath those proxies we'll still allow for polling or listening Tentacles:

  • If the Tentacle is listening, we'll open a TcpClient and connect.
  • If the Tentacle is polling, we'll put the response in a queue, and wait on a wait handle. When the request is picked up and a response is available, the wait handle will complete.

You can think of this as being two layers: a transport layer in which either party can be the TcpClient/TcpListener, and a logical request/response layer that sits on top.

Our new communication stack will be open source (and will build upon the Halibut work I wrote about some time ago), and I'll post more details as it evolves.

Octopus Deploy 2.6 Release

Things are winding down a little bit for a few weeks here at Octopus HQ. We've got a present under the tree for everybody though.

Octopus Deploy 2.6 has been in pre-release for a couple of weeks, we've had some good feedback which we've used to speed a few things up and fix a couple of issues. We're happy that it's fully cooked now and we want to release life cycles and parallel deployments so yesterday we hit the deploy button and 2.6 is officially released.

If you missed the earlier announcement, we have an epic blog post about what's new in 2.6 and of course you can download it here.

Happy deployments!

How we are using SQL Server in Octopus 3.0

In a previous post I announced that we are switching from RavenDB to SQL Server for Octopus 3.0. That post talked about why we were leaving RavenDB, but didn't explain too much about how we plan to use SQL Server. In this post, I want to talk about how were using SQL Server, as well as discuss some minor breaking changes.

We've just finished most of the work involved in porting Octopus 3.0 to SQL Server. We have a suite of automated API tests that install and configure an Octopus server, register some Tentacles, and exercise the application using the REST API. These tests are now passing and running completely against SQL Server:

End to end tests passing

Editions

We are making sure Octopus works on:

  • SQL Server 2005, 2008, 2012, 2014 and above; any edition from Express to Enterprise
  • SQL Azure

To make the getting started experience smooth and easy, the Octopus installer will give you the option to automatically download and silently install SQL Server Express edition, which is free. Of course, you could always connect it to a clustered SQL Server Enterprise server instead, though the licensing costs for other SQL Server editions would be something you'd need to discuss with Microsoft ;-)

High availability

Today, Octopus actually uses a few different data stores:

  • Most data is stored in RavenDB
  • Deployment logs (which are always appended to) are stored on disk because it wasn't possible to append to attachments unless you are Schlemiel the Painter
  • State about in-progress deployments and other tasks was also stored on disk
  • NuGet packages in the built-in repository are stored on disk, with metadata in Lucene.NET indexes

And while we supported using an external (clustered) RavenDB instance, it's not something most customers are really able to set up and manage, so we nearly always use the embedded version of RavenDB. Because we also had data in so many places, we needed to build our own backup and restore features into the product.

For Octopus 3.0, we're going to make sure we have a great high availability story. Most enterprises are already familiar with setting up a clustered SQL Server instance, and have DBA's on site that can help to manage it. So our first design principle will be that everything (nearly) needs to be in SQL Server. Specifically:

  • All the documents we currently store in Raven will go to SQL Server
  • Deployment logs will be compressed (they compress very nicely) and also stored in SQL
  • In progress deployment state: we'll rely on this being in memory (see the breaking change section below)
  • NuGet packages will still be on disk (you'll be able to change where they are stored, and put them on a file share/SAN), but metadata will be stored in SQL

In addition, we're going to make sure that you can set up multiple Octopus Deploy servers, all pointing at the same SQL database/using the same packages directory. Installation wizards and command line tools will make it easy to set up a Siphonophore:

Octopus server load balanced

It won't exactly be web scale, but Stack Exchange have done a good job of demonstrating that you can get pretty far by scaling out application servers and scaling up the database.

Breaking change: There is one scenario that we won't be supporting any longer: restarting the Octopus server during a deployment.

Previously, you could kick off a long running deployment, then shut down the Octopus server, start it again, and there was a pretty good chance it would continue where it left off. I say "chance" because it's impossible to test all the scenarios, and we know some areas where it didn't work and deployments would be left in a weird state where they said they were running but actually weren't. We'll be able to simplify things and get far better performance by removing this feature, and since I don't think it ever fully worked reliably, it should be an OK change. If this affects you, let me know in the comments below!

SQL as a document store

The one feature we loved (and will miss) about using a document database like RavenDB was the ability to store and load large, deep object graphs without a ton of joins. For example, Octopus allows you to define variables, which are key/value pairs that can be scoped to many different fields. Some customers have thousands of these, and we snapshot them every release, so to model this with a traditional relational schema would make things very complicated. And we're never actually going to query against that data, we just need to load it all into memory during deployments.

Instead, we're treating SQL as a document store. Each document type gets its own table, and fields that we query on will be stored as regular columns. But all the fields and deep object graphs that we don't query on are stored as a JSON blob (a nvarchar(max)).

Storing documents in SQL

Since we don't do any joins, we don't need an ORM to help stitch object graphs together. Instead, we're staying close to the metal, essentially using some wrappers around SqlConnection/SqlCommand that use JSON.NET to deserialize the JSON blobs and then set the extra fields. A custom JSON.NET JsonContractResolver excludes properties that are mapped as table columns so the values aren't stored twice.

The only downside to this design is that there are a handful of places where we have to do LIKE %x% queries over tables - e.g., to find all machines tagged with a given role (the list of roles is stored as a pipe-separated nvarchar column on the Machine table). However in all of these cases we expect these tables to be < a few thousand items, so I really don't expect it to matter. If testing shows otherwise, we'll either use full text search or introduce a new table in a CQRS-like index table.

Backup, restore and maintenance

Since all of our data will either be in SQL Server or on a file share (NuGet packages), at this stage I expect to be able to remove our custom backup/restore features and to just rely on SQL Server backups. We'll provide some guidance on how to configure this, and some feedback in the Octopus UI if you have forgotten to do a SQL backup in some time, but in general I think SQL Server's built-in backup/restore features are better than anything we're likely to build.

Migration

The upgrade experience from 2.6 to 3.0 will be straightforward: you'll install 3.0, select/create a SQL Server database to use, and then choose an Octopus 2.6 backup to import. We'll convert the data as needed and then you'll be up and running in no time. It will feel much more like upgrading between 2.5 and 2.6 than upgrading from 1.6 to 2.0.

So far we've done nearly all of the conversion to SQL Server and haven't had to make any API changes, so any code against our 2.X REST API will work against 3.0.

Testing

We collect (opt-in) usage statistics, and there are some big Octopus installations out there - 300+ projects, 1000+ machines, with over 20,000 deployments. We'll be using this data to simulate similar environments and to ensure we don't release anything that is slower than what we already have.

We'll start by running our end-to-end tests and comparing the current 2.6 builds with the upcoming 3.0 builds to ensure that none of our current operations are any slower on smaller data sets. Then we'll move on to load testing to ensure that we can handle at least 5x larger installations than we currently have without crazy hardware requirements.

If anyone's interested in seeing some of these metrics, let me know in the comments and I'll do a third post in this series :-)

TeamCity 9 plugin compatibility

We're big fans of TeamCity here at Octopus Deploy, so we're as excited as everybody else about their recent release of TeamCity 9.

Everybody is talking about the ability sync your project build settings with your version control system. It's something we've been thinking about a lot lately too so it's going to be interesting to see what sort of takeup the feature has.

The big question we've been getting asked over the last few days of course is compatibility between our TeamCity plugin and the new release from JetBrains.......

The answer, I'm happy to say, is...YES!

I've installed the plugin with a brand new, fresh out of the oven TeamCity 9 install and everything works just fine.

Happy building!

In Octopus 3.0, we're switching from RavenDB to SQL Server

Early beta versions of Octopus used SQL Server with Entity Framework. In 2012, just before 1.0, I switched to using RavenDB, and wrote a blog post about how we use the Embedded version of RavenDB.

For over two years we've been developing on top of RavenDB. In that time we've had over 10,000 installations of Octopus, which means we've been responsible for putting RavenDB in production over 10,000 times. And since most customers don't have in-house Raven experts, we've been the first (only) line of support when there are problems with Raven. We haven't just been kicking the tyres or "looking at" Raven, we bet the farm on it.

For Octopus 3.0, we are going to stop using RavenDB, and use SQL Server instead. Understandably many people are interested in "why". Here goes.

First, the good

RavenDB has a great development experience. Compared to SQL + EF or NHibernate, you can iterate extremely fast with RavenDB, and it generally "just works". If I was building a minimum viable product on a tight deadline, RavenDB would be my go-to database. We rewrote nearly all of Octopus in 6 months between 1.6 and 2.0, and I don't think we could have iterated that quickly on top of SQL + EF.

The bad

We handle most support via email/forums, but when there are big problems, we escalate them to a Skype/GoToMeeting call so we can help the customer. Usually that's very early in the morning, or very late at night, so minimizing the need to do them is critical to our sanity.

What's the cause of most of our support calls? Unfortunately, it's either Raven, or a mistake that we've made when using Raven. And it's really easy to make a mistake when using Raven. These problems generally fall in two categories: index/data corruption issues, or API/usage issues.

Above all else, a database needs to be rock-solid and perform reliably. Underneath Raven uses ESENT, and we've generally not lost any data from the transactional side of Raven. But indexes are based on Lucene.NET, and that's a different story. Indexes that have broken and need to be rebuilt are so common that for 1.6 we wrote a blog post explaining how people can reset their indexes. We sent this blog post to so many people that in 2.0 we built an entire feature in the UI to do it for them.

Repair RavenDB

When I said we'd never lost the transactional data, that's not quite right. It's really easy in RavenDB to add an index that causes big, big problems. Take this:

  Map = processes => from process in processes
                     from step in process.Steps
                     select {...}
  Reduce = results => from result in results
                      group result by ....

You can write this index, and it works fine for you, and you put it into production. And then you find a customer with 10,000 process documents, each of which have, say, 40 steps.

While Raven uses Lucene for indexing, it also writes index records into ESENT. I don't know the internals, but there are various tables inside the Raven ESENT database, and some are used for temporarily writing these map/reduce records. For every item being indexed, it will write a huge number of records to these tables. So we get a support issue from a customer: they start Octopus, and their database file grows at tens or hundreds of MB per second, until it fills up the disk. The database file becomes so large that they can't repair it. All they can do is restore from a backup. When we finally got a copy of one of these huge data files, and explored it using some UI tools for ESENT, these tables contained millions upon millions of records, just for 10,000 documents.

The RavenDB team realised this was a problem, because in 3.0 they added a new feature. If a map operation produces more than 15 output records, that document won't be indexed.

I mean, just read that paragraph again. You write some code, test it, and it works fine in development. You put it in production and it works fine there too, for everyone. And then you get a call from a customer: I just added a new process, and it's not appearing in the list. Only after many emails and a support call do you realise that it's because Raven decided that 15 is OK and 16 is not, and the item isn't being indexed. Your fault for not reading the documentation!

"Safe by default" is so painful in production

Raven has a "safe by default" philosophy, but the API makes it so easy to write "safe" code that breaks in production. For example:

session.Query<Project>().ToList();

Put this in production and you'll get a support call: "I just added my 129th project and it isn't showing up on screen". In order to protect you from the dreaded "unbounded result set" problem, Raven limits the number of items returned from any query. Be thankful it wasn't this:

DeleteExcept(session.Query<Project>().Where(p => !p.KeepForever).ToList())

Unbounded result sets are bad, sure. But code that works in dev, and in production, until it suddenly behaves differently when the number of records change, is much worse. If RavenDB believes in preventing unbounded result sets, they shouldn't let that query run at all - throw an exception when I do any query without calling .Take(). Make it a development problem, not a production problem.

You can only do 30 queries in a session. Result sets are bounded. Only 15 map results per item being mapped. When you work with Raven, keep these limits in your mind *every single time you interact with RavenDB, or you'll regret it.

These limits are clearly documented, but you'll forget about them. You only become aware of them when something strange happens in production and you go searching. Despite two years of production experience using Raven, these opinions still bite us. It frustrates me to see posts like this come out, advocating solutions that will actively break in production if anyone tries them.

Conclusion

RavenDB is great for development. Maybe the problems we're experiencing are our fault. All databases have their faults, and perhaps this is a case of the grass is always greener on the other side. Switching to SQL Server might seem like a step backwards, and might make development harder, but at this point I do feel like we will have less problems in production with SQL Server. It has been around for a long time, and the pitfalls are at least well known and predictable.

That's enough about why we're leaving RavenDB. Next week I'll share some details about how we plan to use SQL Server in Octopus 3.0.

(*) You can disable the unbounded result set protection thing by specifying unlimited items to be returned, if you know where to turn it off. But you still have to explicitly call .Take(int.MaxValue) every single time you write a query.

%TEMP% has different values for a Windows Service running as Local System

You probably already know that Environment variables can be defined at either machine scope, or user scope. The value at the user scope typically overrides the value defined at machine scope.

Environment variables

However, there's a special case for Windows Services that run as the SYSTEM account. Given the following Windows Service:

public partial class Service1 : ServiceBase
{
    public Service1()
    {
        InitializeComponent();
    }

    protected override void OnStart(string[] args)
    {
        File.WriteAllText("C:\\Temp\\Service.txt", 
            "Temp:        " + Environment.GetEnvironmentVariable("Temp") + Environment.NewLine +
            "Temp (User): " + Environment.GetEnvironmentVariable("Temp", EnvironmentVariableTarget.User) + Environment.NewLine);
    }
}

When the service runs as my user account, I get what I'd expect:

Temp:        C:\Users\Paul\AppData\Local\Temp
Temp (User): C:\Users\Paul\AppData\Local\Temp

However, run the service as the built-in SYSTEM (Local System) account, and you get different behavior:

Temp:        C:\WINDOWS\TEMP
Temp (User): C:\WINDOWS\system32\config\systemprofile\AppData\Local\Temp

It appears that for Windows Services that run under the SYSTEM account, even though there's a user-specific environment variable, a different %TEMP% is used.

This caused a bug in yesterday's 2.6 pre-release because we added a feature to automatically update environment variables prior to each script run (in case you've changed environment variables, and don't want to restart the Tentacle windows service). Of course, no good deed goes unpunished :-)

I can't find any documentation on this feature, but environment variables are inherited by processes from their parent. Services are owned by services.exe, which is owned by wininit.exe. Using Process Explorer, wininit.exe's environment variables set TEMP to C:\Windows\TEMP. My guess is this is probably a backwards compatibility feature for old Windows Services that relied on using C:\Windows\TEMP.

(We'll release a patch to 2.6 tomorrow with a fix for this)

What's new in Octopus Deploy 2.6

Octopus Deploy 2.6 is now in pre-release! For those who like to live on the edge, you can Download the Octopus Deploy 2.6 pre-release. And who wouldn't want to live on the edge when this release contains so many new features? Here are the highlights:

  • Lifecycles to control promotion and automate deployments
  • Automatic release creation from NuGet push
  • Running steps in parallel
  • Up to 5x faster package uploads
  • Skipping offline machines

Lifecycles

This heading just does not have enough fanfare for this feature. Imagine balloon's springing out, trumpeters trumpeting and confetti cannons making a mess everywhere.

Okay I will stop, but I do love this feature!

Lifecycles main

Lifecycles let you specify and control the progression of deployments to environments. You not only have the ability to order environments for deployment but you can:

  • set environments to auto deploy when they are eligible for deployment
  • gate your workflow to be sure that N QA environments have been deployed to before moving on
  • group multiple environments into a single stage
  • deploy to more than one environment at a time

Yep, deploy to more than one environment at a time!

A Lifecycle consists of phases and retention policies. Let's start with phases.

Lifecycle Phases

Lifecycle phases

A Lifecycle can consist of many phases. A phase can consist of many environments. Each phase will allow you to add environments to it. You can gate each phase, to be sure that N many environments have been released to before the next phase is eligible for deployment.

Lifecycle automagic

When selecting which environment to add to a phase you have the choice of if they will be a manually release or auto release. If they are set to auto release, when they get to their phase of the deployment chain, they will begin deployment.

Lifecycles and Retention Policies

Lifecycles have their own retention policies. Each has an overall retention policy that each phase will inherit. However you can overwrite this for each phase.

Lifecycle Retention Policy 1

This will mean for development where you have 1300 releases a week, you can have a very strict retention policy set to delete all but the last 3 releases. But for production you can keep everything forever. Or somewhere in the middle if your needs aren't so extreme

Lifecycles and Projects

Lifecycles are assigned to projects via the Process screen.

Lifecycles project process

You might notice that your Projects Overview screen has had a bit of an overhaul.

Lifecycles project overview

It will now display your most recent releases, where they are at per environment, and provide any deployment or promotion buttons. It allows you to see the most current, and previous deployments at a glance. Solid green being your most recent deployment, medium faded green as your previous deployment, and light faded green as all others.

Lifecycles, Releases, Deployments, and Promotions

Lifecycles releases

The release page now gives you a graphical tree of where a deployment is currently, which phase and what's deploying. You may notice a few things here. The deploy/promote button got a bit smarter. It knows whats next in the chain. It also allows you to deploy to any environment that has already been deployed to.

Lifecycles two at once

You can now release to multiple environments at the click of one button. Yep.

Lifecycles multiple environments

Or you can use this select box, and choose them all!

Lifecycles smart promote

And when you have finished a deployment, the promote button knows what's next and gives you the option to promote to the next environment.

Lifecycle deployment

The deployment screen got a little simpler too. Much easier to find the deploy now button. But don't worry everything is available under Advanced, and if you are so inclined you can tell it to remember to show the advanced settings all the time.

Lifecycles and Blocking Deployments

If you have a bad release that has just done some bad stuff (tm) to your QA server, you might want to block that release from being released further down the chain until the issue is resolved. Now you can block a deployment.

block deployment

Step 1 block the deployment for reasons.

show blocked deployment

On your release screen you can see that the promote button has vanished and your Lifecycle deployment tree has a red icon for those environments it cannot promote to.

name

You will also now see a warning marker on your overview, and will no longer see promotion buttons for that release. In fact all promotion buttons are gone for it. You can only at this point deploy to environments that have already been deployed to. Unblocking when the issue is resolved will give you full access to deploy the release.

Lifecycles and Automatic Release Creation

Yes, there is still more!

Lifecycles automagic settings

On a project process screen, you can define the name of a package, that when pushed or uploaded to the internal repository will automatically create a release for you.

lifecycles automagic create

And if you have your first environment in the Lifecycle setup to automatically deploy, this means you can push a NuGet package to the internal repository and have it automatically create a release and deploy it! We are looking at you TFS users!

As you can see Lifecycles is a feature that has it's hands in many areas of Octopus, it's a large feature, that we are very proud of. We have used this opportunity to try to listen to your feedback and suggestions in UserVoice to to add in more value. We really hope you like it as much as we do!

Run Steps in Parallel

Another feature in 2.6 allows you to setup multiple project steps to run in parallel.

Project step trigger

You can select on a project step for it to run in parallel with the previous step.

project step together

The process page has been updated to show these steps grouped together. If they run on the same machine, they will still queue unless you configure the project to allow multiple steps to run in parallel

Retention Policies have moved

As seen above, retention policies have moved into Lifecycles. You will no longer find a Retention Policy tab under Configuration. They are also not able to be set for Project Groups any more. This leaves setting the retention policy for the internal package repository.

repository retention settings

This has been moved to where the packages live, under Library -> Packages.

Package Upload Streaming

In 2.6 when Octopus downloads a package and then sends it to the Tentacles, it will now do so via streaming. We have seen a 5x speed increase. This will also alleviate some of the overhead memory that Tentacle was using to store the package chunks before saving.

SNI Support

As well as fixing up some issues with SSL bindings, we added SNI support.

SNI support

Skip Offline Machines

Currently, when doing a deployment to very large environments, offline machines can get in your way. We now give the ability to continue with the deployment but skip the offline machines.

show offline machines

We now show offline machines (displayed in red) on the deploy screen. This will allow you to go back and check the machines connection. Or you can use the "ignore offline machines" feature.

ignore offline machines

This will automatically list all machines but the offline machines.

This ends the tour of what's new in 2.6. We have only mentioned the big features in this release, but there were quite a few smaller changes and bug fixes made, so please check the release notes for more details on these smaller items. We hope you are excited about Lifecycles as we are!

Download the Octopus Deploy 2.6 Pre-release now!

Invoking an executable from PowerShell with a dynamic number of parameters

Calling an executable from PowerShell is easy - most of the time, you just put an & in front. To illustrate, let's take this C# executable:

static void Main(string[] args)
{
    for (int i = 0; i < args.Length; i++)
    {
        Console.WriteLine("[" + i + "] = '" + args[i] + "'");
    }
}

If we call it like this:

& .\Argsy.exe arg1 "argument 2"

We get:

[0] = 'arg1'
[1] = 'argument 2'

PowerShell variables can also be passed to arguments:

$myvariable = "argument 2"
& .\Argsy.exe arg1 $myvariable

# Output:
[0] = 'arg1'
[1] = 'argument 2'

Note that the value of $myvariable contained a space, but PowerShell was smart enough to pass the whole value as a single argument.

This gets tricky when you want to conditionally or dynamically add arguments. For example, you might be tempted to try this:

$args = ""
$environments = @("My Environment", "Production")
foreach ($environment in $environments) 
{
    $args += "--environment "
    $args += $environment + " "
}

& .\Argsy.exe $args

However, you'll be disappointed with the output:

[0] = '--environment My Environment --environment Production '

The right way

The way to do this instead is to create an array. You can still use the += syntax in PowerShell to build the array:

$args = @() # Empty array
$environments = @("My Environment", "Production")
foreach ($environment in $environments) 
{
    $args += "--environment"
    $args += $environment
}
& .\Argsy.exe $args

Which outputs what we'd expect:

[0] = '--environment'
[1] = 'My Environment'
[2] = '--environment'
[3] = 'Production'

You can also mix regular strings with arrays:

& .\Argsy.exe arg1 "argument 2" $args

# Output:
[0] = 'arg1'
[1] = 'argument 2'
[2] = '--environment'
[3] = 'MyEnvironment'
[4] = '--environment'
[5] = 'Production'

Edge case

There's is a very odd edge case to what I said above about passing a single string with all the arguments. Take this example, which is similar to the one above:

$args = "--project Foo --environment My Environment --environment Production"
& .\Argsy.exe $args

# Output: 
[0] = '--project Foo --environment My Environment --environment Production'

To make it work as intended, just put a quote around the first argument, and the behaviour changes completely! (The backticks are PowerShell's escape characters)

$args = "`"--project`" Foo --environment My Environment --environment Production"
& .\Argsy.exe $args

# Output: 
[0] = '--project'
[1] = 'Foo'
[2] = '--environment'
[3] = 'My'
[4] = 'Environment'
[5] = '--environment'
[6] = 'Production'

The behavior doesn't change if the first argument isn't quoted:

$args = "--project `"Foo`" --environment My Environment --environment Production"
& .\Argsy.exe $args

# Output: 
[0] = '--project Foo --environment My Environment --environment Production'

Ahh, PowerShell. Always full of surprises!

Dynamically setting TeamCity version numbers based on the current branch

When you are using TeamCity to build a project with multiple branches, it's desirable to have different build numbers depending on the branch. For example, instead of simple TeamCity build numbers like 15, 16, and so on, you might have:

  • Branch master: 1.6.15
  • Branch release-1.5: 1.5.15 (major/minor build from branch name)
  • Branch develop: 2.0.15 (different minor build)
  • Branch feature-rainbows: 2.0.15-rainbows (feature branch as a tag)

Here's how it looks:

TeamCity builds with build numbers based on the branch

Handling a branching workflow like GitFlow, and using these version formats, turns out to be pretty easy with TeamCity, and in this blog post I'll show you how. Your own versioning strategy is likely to be different, but hopefully this post will get you started.

Background

First, there are two built-in TeamCity parameters that we care about:

  • build.counter - this is the auto-incrementing build counter (15 and 16 above)
  • build.number - this is the full build number. By default it is %build.counter%, but it can be more complicated

The format of build.number and value of build.counter is defined in the TeamCity UI:

Build number and build counter in TeamCity

However, you can also set it dynamically during the build, using service messages. That is, your build script can write the following text to stdout:

##teamcity[buildNumber '1.1.15']

This will override the build number, and the new value will then be passed to the rest of the steps in the build.

Putting it together

Depending on whether the branch name is master or develop, we will use different major/minor build numbers. To do this, we're going to define two parameters in TeamCity. These need to be "system" parameters in TeamCity so that they are available to build scripts.

Adding the major/minor build number parameters

To dynamically set the build number based on the branch name, I'm going to add a PowerShell script step as the first build step in my build:

Using a PowerShell script build step to set the build number

Finally, here's the PowerShell script:

# These are project build parameters in TeamCity
# Depending on the branch, we will use different major/minor versions
$majorMinorVersionMaster = "%system.MajorMinorVersion.Master%"
$majorMinorVersionDevelop = "%system.MajorMinorVersion.Develop%"

# TeamCity's auto-incrementing build counter; ensures each build is unique
$buildCounter = "%build.counter%" 

# This gets the name of the current Git branch. 
$branch = "%teamcity.build.branch%"

# Sometimes the branch will be a full path, e.g., 'refs/heads/master'. 
# If so we'll base our logic just on the last part.
if ($branch.Contains("/")) 
{
  $branch = $branch.substring($branch.lastIndexOf("/")).trim("/")
}

Write-Host "Branch: $branch"

if ($branch -eq "master") 
{
 $buildNumber = "${majorMinorVersionMaster}.${buildCounter}"
}
elseif ($branch -eq "develop") 
{
 $buildNumber = "${majorMinorVersionDevelop}.${buildCounter}"
}
elseif ($branch -match "release-.*") 
{
 $specificRelease = ($branch -replace 'release-(.*)','$1')
 $buildNumber = "${specificRelease}.${buildCounter}"
}
else
{
 # If the branch starts with "feature-", just use the feature name
 $branch = $branch.replace("feature-", "")
 $buildNumber = "${majorMinorVersionDevelop}.${buildCounter}-${branch}"
}

Write-Host "##teamcity[buildNumber '$buildNumber']"

Now that %build.number% is based on the branch, your TeamCity build has a consistent build number that can then be used in the rest of your build steps. If you are using OctoPack, for example, the build number can be used as the value of the OctoPackPackageVersion MSBuild parameter so that your NuGet packages match the build number.