This week we've been working on the migration tool to import Octopus 1.6 databases into Octopus 2.0. Since 2.0 changed a lot, our approach has been to have Octopus 2.0 run side-by-side with Octopus 1.6. The migration process will be:
- Install Octopus 2.0
- Open the Octopus Manager UI
- Click "Import from 1.6"
- Select your Octopus 1.6 backup
In the background, we'll create a new RavenDB database, and import your 1.6 backup. Then, we'll stream the data from the 1.6 database to the 2.0 database, converting the documents on the fly. This means that after the migration, you can still run Octopus 1.0 and 2.0 side-by-side.
A few customers were kind enough to send us their 1.6 backups to test against which was very helpful. Expect to see the 1.6 migrator in the Octopus 2.0 RC build coming soon.
This Saturday I'll be doing a presentation at DDD Brisbane.
Automated deployment using TeamCity and Octopus Deploy
Agile software delivery revolves around getting working software in front of people early, and that's easy when you're delivering a small demo application. But when deployments involve delivering multi-tiered solutions to multiple web, application and database servers in production and pre-production environments, deployment itself can be a risky, heart-pounding and time-consuming process.
In this talk, I'll look at streamlining the delivery of .NET applications all the way from source control to pre-production and production environments. At the push of a button (or a source code commit), we'll take code, compile it, run unit tests, deploy it to multiple servers in a test environment, run smoke tests, and then promote it to production. I'll show how TeamCity and Octopus Deploy can work together to put working software into the hands of users, faster.
I'm excited about this presentation because it's the first time we'll be demonstrating Octopus 2.0 in public! The session will be recorded and I'll post a link to it here when it is available.
We're also proud to be silver sponsors for DDD Brisbane this year.
I'm excited to announce that, starting today, you can download the Octopus Deploy 2.0 public beta builds from a new GitHub repository that we've set up to manage the public beta:
This version of Octopus has been 6 months in the making. We've touched nearly every part of the code in order to implement a lot of new features, and build a foundation upon which we can continue to grow.
Over the coming weeks I'll be blogging more about the new features and changes we've made in 2.0, but for now if you have a spare VM laying around and some time to spare, why not give it a try and see what's new? Happy deployments!
Octopus 2.0 isn't just a couple of new features; it's practically a ground-up rewrite. Right now we're at a point where most of the functionality works fine, but there are some missing pieces (mostly around installation/configuration) and usability quirks, so we've been doing "hand-held" preview releases to a few users to start getting feedback.
Going forward, the roll out plan for 2.0 looks something like this:
Right now we're tidying up a few bugs to do a final hands-on preview release. Then we'll move to finishing any items needed for a private beta (beta 1).
Beta 2 will be the first public release, and by then we hope to have something that is ready for day-to-day use, even if there are a few small bugs and no migration tool. We'll give it some time to settle and mature and build a migration tool from 1.X before we declare it "production ready".
I'll leave you with a screenshot from the new Tentacle setup wizard:
At Octopus Deploy, our mission is to improve the software industry as a whole by helping teams to deliver working software to production in a way that is more reliable and less risky. From our past experiences, we think that automation plays an important part in helping to achieve this. By automating software deployments, we believe:
- Deployments become less risky and more reliable
- Deployments become faster
- These combine to encourage and enable teams to deploy more often
- Deploying more often leads to a tighter feedback cycle
So far this has been based on our own qualitative experiences prior to founding Octopus Deploy, and from the occasional email from customers. We'd love to get some more numbers around this, though.
If you can spare a minute to answer a handful of multiple choice questions about your experiences with automated deployments, we'd really appreciate it! Your experiences will help others to build a case for deployment automation within their organization.
In Octopus we take a strong view that deployments should be repeatable. When you prepare to deploy a project using Octopus, you take some time to define your deployment process. In Octopus, this is represented by a set of deployment steps that you define on the Steps tab.
I've noticed a couple of common mistakes that people make when it comes to defining steps. To help, here are two things you can do to reduce the need to have to duplicate your deployment steps.
Tip 1: Use the same roles in all environments
When you create a package or script step, you specify the roles that the step is deployed to:
People sometimes assume that when I select the
web-server role above, the step will be run against all machines in that role regardless of the environment I deploy to. That isn't the case. When I deploy to my Development environment, the step will only be run against
web-server machines in the Development environment. When I deploy to Production, the step will only be run against machines in the
web-server role in the Production environment.
If your role names end with "prod" or "uat" or "dev", it's quite likely that you aren't using roles properly.
There's a theory that when it comes to software, if the user assumes it must work a certain way, and it doesn't, then the software is at fault and not the user. I think that's largely true. In this case I suspect it's because we don't do a good job from a UX point of view of making it clear what will happen during deployment. If you have any suggestions on how we can make this clearer, please leave a comment below.
Tip 2: Use variables to account for differences between environments
When defining a step, many fields allow you to reference variables. You can tell whether a field supports it by looking for the hash button next to a field.
Suppose I need to deploy an Azure cloud service package to two different cloud services in Azure. I could go and create two different steps, and select a different environment for each. A better way, however, is to define variables scoped to each environment.
Then, I can refer to that variable when defining a step:
Hopefully these two tricks will help you to reduce the number of deployment steps you need to define.
For a while now, Octopus has supported the ability to reference variables when defining step properties. For example, when defining an FTP step, we ask you what root directory we should upload files to.
You can type a raw value:
Or you can reference a variable:
This is a powerful feature because the variables you reference can be scoped to different environments. This way you can use a different root directory path per environment.
One limitation of this feature however is that it only worked for values that are presented as text boxes. For values that are usually presented as checkboxes, or radio buttons, or select boxes, we didn't have a way to let you "bind" these fields to an expression. For Octopus 2.0 I want to change that.
This is what I have so far. We'll still present the fields as checkboxes by default:
But if you need to do something more advanced, you can click the "binding light" button on the right, and bind it to a custom expression:
Here's the JSFiddle embedded:
Nearly all step properties are going to be bindable like this. Checkboxes, select boxes, auto completes, radio groups, and anything else, will all have a binding light that allow you to enter a custom expression.
Could the UX be improved? Leave a comment in the box below.
We've got a couple of related suggestions on UserVoice that I'd like to address in Octopus 2.0:
I want to use this blog post to share how we're going to address these needs. Octopus 2.0 is going to support a number of different scenarios relating to encryption.
- Encrypting the database
The RavenDB database in Octopus needs to be encrypted to allow customers to 'tick off' Octopus for use in environments which require any configuration information to be encrypted on disk. For example, customers in PCI-DSS environments.
- Secure/sensitive variables
Some variables created in the Octopus UI may also need to be encrypted. For example, a customer might create a variable to represent an administrator's password for use when connecting to a database. This extends beyond on-disk encryption - these values shouldn't be returned by the REST API, nor should they ever appear in activity logs. They should be "write-only".
- Storage of sensitive internal Octopus settings
For example, the private key for the Tentacle X.509 certificates are currently stored in the registry, encoded using Base64. These values should be encrypted too.
To make this work, we're going to rely on two levels of encryption. A "master key" will be used with AES128 to symmetrically encrypt values. The master key itself will be encrypted asymmetrically using DPAPI.
When Octopus first starts, it will generate a random string to use as a master key. This master key will then be encrypted using DPAPI and stored in the Octopus configuration file.
Octopus will prompt you to back up your master key whenever you open the Octopus admin tool. Until you click something to the effect of 'I have backed up my master key', you'll continue to be prompted every time you open the admin tool. To back up your key, we'll simply expose the master key as a Base64 encoded representation of the unencrypted key. You can print it or paste it into Notepad and save it wherever you like.
Encrypting the database
RavenDB comes with a bundle that enables encryption for the database. When we run Raven in embedded mode, we'll automatically enable this bundle, and use our master key as the key for Raven's encryption bundle. This ensures that all documents and indexes are encrypted. Since the bundle needs to be enabled when the Octopus database is created (it can't be turned on later), this isn't going to be an opt-in/opt-out feature. It will be out of the box.
In the scenario where customers use an external RavenDB database, it will be up to them to enable this bundle and to manage their own key for it.
The following, however, will not be covered with Raven's encryption bundle:
- Raven attachments
- Log files
For attachments, we'll encrypt and decrypt these ourselves using AES128 and our master key. We won't be encrypting log files however. Storing the logs in plain text has a lot of value when it comes to debugging, and since we'll prevent sensitive variables from appearing in them (see below) there should be no reason to encrypt them.
When you create a variable, you'll be able to tick a box that makes it 'secure'. A secure variable cannot be read once it has been written; it will always appear as '**' in the UI, and the value won't be available from the REST API. It will, however, be made available as a variable when sent to Tentacles.
The value of these secure variables will be stored encrypted using the master key and AES128. Variables that aren't marked as secure will simply have their values stored as plain text.
When processing any activity logs, we're going to take inspiration from the way TeamCity deals with password parameters: the values of any secure variables will automatically be masked. For example, if a user accidentally wrote a Deploy.ps1 script like this:
Write-Host "Password is $MyPassword"
Octopus would see the secure value appear in the log, and would automatically replace it with "*" before the log entry is sent over the wire or persisted. In the UI, you'll see:
Password is ***************
This will simply be done using
string.Replace(). Of course, this does mean that a user could write a script that generates a random values and then look for the '*' to figure out which one was a password, and if passwords are a simple as 'hello' the masks might show up in the wrong place and thus give themselves away, but on the whole this feature is meant to be an extra layer of protection and not 100% fool proof.
Backup and restore
Backups are currently performed using an 'import' and 'export' of documents via RavenDB's Smuggler API. The file it produces is just a list of JSON documents with GZIP compression.
Our backup file will need to be encrypted too, so we'll take the backup file from smuggler, add the activity logs and any other files we need to back up, and compress them all into a
System.IO.Packaging package. Then we'll apply encryption to that, using the master key.
When you restore from a backup, you'll need to enter the master key, which should have been backed up/printed/saved when Octopus was set up. We'll then set the master key in your configuration file and perform the restore.
Our solution to these problems is going to involve a few dimensions. We're going to automatically encrypt all documents in Raven via the Raven bundle. We're also going to allow for variables to be marked as "secure", which impacts how they are presented in the UI.
All of our encryption is going to rely on AES128 using a randomly generated master key, and the master key itself will be stored using DPAPI, so if attackers manage to see your configuration file they still won't be able to read the key unless they have code running on your Octopus server. The only downside to this change is the need to back up your master key, which we'll try our best to make a nice user experience.
This feature is currently in the planning stages, so if you're in an environment where these features are important, I'd love to get your feedback in the comments below.
In "UI design for eventual consistency", I wrote about how we commonly got bug reports like this:
I went to the "add machine" page, entered the machine information, clicked save, but when I went to the environments page, my machine wasn't listed. A few seconds later I hit refresh and there it was.
This was a result of Raven's use of asynchronous indexes. As a developer using RavenDB, you're supposed to embrace eventual consistency. There are common patterns to deal with this, for example:
- you could accept the 'add machine' request as a command, and queue it to be processed, then redirect the user when the processing is done; or,
- you could store the new machine in an intermediate cache, and include it when querying the list
And that makes sense when your operations are "submit a request to refund this bank transaction". But when you have a dozen different document types and you need a bit of CRUD and you aren't web scale, these patterns are, frankly, overkill.
So in my naivety, Octopus 1.0 was scattered with:
.Customize(x => x.WaitForNonStaleResultsAsOfNow())
This of course had problems, and for Octopus 2.0 I resolved to embrace the eventual consistency nature of Raven.
At first, I went down the path of having our REST API return a flag indicating whether results of a query were stale. Our UI would then render a message to notify the user, or automatically refresh after a second to try and get the non-stale results. While it made the UI feel faster, it was a bit jarring.
However when building some command line tools to use the API, I realised that most of the time, these tools would rather wait for non-stale results. So that became a flag (e.g.,
/api/environments?nonStale=true) that was provided as part of a URI template. This caused some debate; after all, should clients even be able to request such a thing?
You want to write this record, then in the same request query that index with WaitForNonStaleResultsAsOfLastWrite, then redirect the user to the list after the index is synchronized. This avoids making the common operation, reads, from always waiting, and makes the uncommon operations, writes, to have to wait.
I'd never seen this suggestion before, but I think it's brilliant. I'm thinking about it as "perceptual consistency", or perhaps, "per-user consistency".
What it translates to is that in the Octopus REST API, when you perform a PUT/POST/DELETE, we'll wait for the indexes to be non-stale before returning success (for a little while at least). So you may see a slight pause when you hit "create machine". But when you are redirect and perform a GET, that request won't wait for non-stale results. And you won't care, because by the time you're redirected, the index is up-to-date enough to include the machine you just added. Brilliant!
We'll still provide the non-stale flag on query results to indicate whether the results of your GET request are stale, but there's no way to tell the API whether to wait for non-stale results or not anymore. And from a UI point of view you'll always see results that look consistent to you.