Octopus Deploy has had a RESTful HTTP API for some time, available at:

http://<your-octopus-server>/api

The API was built primarily to power Octo.exe, a command line tool for creating and deploying releases. Most of the existing API is read-only, with the exception of machines, releases and deployments. When we introduced new features to the Octopus UI, they didn't always make it into the API; it's fair to say the API was really a second class citizen.

For Octopus Deploy 2.0, we're going to make some big changes to our UI, which I blogged about in UI design for eventual consistency. I want to make sure that Octopus feels fast and fluid even on very large installations. In addition, we often get feature requests to be able to do things in the API that we currently can only do via the UI.

So for the 2.0 release, we're going to make our API a first class citizen. The UI will leverage the API; this means that most of the interactions in the UI will make asynchronous requests via the API rather than posting directly to MVC controllers.

Since the API is getting so much attention, we're also going to document it properly as we rebuild it. To that end I've created a project on GitHub:

OctopusDeploy-API on GitHub

The project is fairly sparse currently, but you'll find a section on authentication and the way we use links to navigate the API.

If you're building on top of the Octopus API, or plan to, I'd suggest following this repository and providing feedback through issues/pull requests in the documentation.

We just released Octopus Deploy 1.6, which you can get from the downloads page. This release is going to be the last major 1.X release; from now on we'll be focusing our attention on Octopus Deploy 2.0. I'll post some more thoughts on what will be in 2.0 soon, but for now let's look at what's new in 1.6.

Direct package download from Tentacles

I blogged our plans for this last month:

The second change would be an option on the package step settings to have Tentacles download packages directly. Instead of Octopus downloading the packages, Octopus would instruct Tentacles to download the packages. The downloads would all happen before the deployment steps are run (again, so that we aren't waiting to copy a package between deployment steps), but they would go direct to the NuGet server.

When creating a package step, you can opt in for this by selecting from the radio buttons below:

Improved release creation screen

We also implemented some planned changes to the create release screen.

Proxy servers

We added the ability for Octopus and Tentacle to use proxy servers, including proxies that use basic and integrated authentication. You can specify the proxy server in the administration tool:

Step variables

Previously, inside of a package step you could use a number of predefined variables:

Octopus.Step.Name
Octopus.Step.Package.NuGetPackageId
Octopus.Step.Package.NuGetPackageVersion

However these variables weren't available inside email or manual steps, since there's no package. This caused problems if you wanted to send an email containing the NuGet version number of a previous step.

Now you can access the variables by index:

Octopus.Step[0].Name
Octopus.Step[0].Package.NuGetPackageId
Octopus.Step[0].Package.NuGetPackageVersion

The index is 0-based and numbered according to the order the steps are executed in.

1.6 also included a number of other bug fixes and changes which you can read more about in the release notes. Happy deployments!

Back when I first started building Octopus Deploy, I searched iStockPhoto for Octopus images, and came across this image:

The colors in that image became the color scheme for the first beta releases of Octopus Deploy:

I needed a smaller logo that I could use in the top corner, though, so I asked my sister to design a logo. That logo became the logo we've used since beta:

I felt the logo was good enough for a version 1.0 release and I liked the smiley face. In fact, a customer even suggested having the logo "frown" when a release failed or the server crashed, so we added that a few releases ago.

As nice as the smiley face is, however, it always felt a little too cartooney for a "professional" product, and it doesn't scale too well at small sizes. Octopus is growing up, so we engaged Corey Ginnivan to give the logo a makeover. Without further adieu, I'm proud to announce the new Octopus logo!

In this post, I want to share some changes that I'm considering to the deployment workflow that Octopus Deploy uses.

Currently in Octopus, any deployment consists of three stages:

1. The Octopus Server downloads all necessary packages from the NuGet feed
2. The Octopus Server securely uploads the packages to the required Tentacles
3. The deployment steps run (packages are extracted and configured)

This can be summed up in the following diagram:

We do this for a simple reason: packages can be big, and it can take a long time to upload them. We don't want to deploy one package (for example, a DB schema change), then wait 20 minutes for the upload to complete of the next package before continuing. Instead, we upload everything, then we configure everything.

But there are some downsides to this architecture, which I'll discuss along with some solutions I'm considering.

Pre-upload steps

The first problem with this architecture is that it's not currently possible to run any steps (including ad-hoc PowerShell scripts) before the packages are uploaded. This prevents using scripts to do useful tasks like establishing a VPN connection. It would be nice to insert steps that are run before the download/upload activity happens.

Bandwidth

Since Octopus pushes a copy of each package to the relevant Tentacles, there's potentially a lot of wasted bandwidth. For example, imagine a scenario where the Octopus server is in a local network, and it is deploying to many machines in the cloud. The same package will be pushed over the internet many times:

An alternative that would save bandwidth is to replicate the packages to a NuGet server in the cloud and to have each Tentacle pull the package from it:

This architecture could also work nicely when deploying to machines in different data centers using DNS or a distributed file system to ensure the machines pick the NuGet server closest to them.

Improving the deployment process

To support these scenarios, I'm considering two changes to the deployment process.

The first will be the concept of pre-package upload steps, which would be run before NuGet packages are downloaded. You could add PowerShell steps here to establish a VPN connection or perform other tasks before the package download process begins.

The second change would be an option on the package step settings to have Tentacles download packages directly. Instead of Octopus downloading the packages, Octopus would instruct Tentacles to download the packages. The downloads would all happen before the deployment steps are run (again, so that we aren't waiting to copy a package between deployment steps), but they would go direct to the NuGet server.

Hopefully these changes will make it easier to handle deployments in distributed environments. I'd love to get your feedback on the proposed changes in the comments box below.

When deciding whether to invest in automating a task, we often weigh up the time it takes to set up the automation versus the time the automation is expected to save. But automation brings another benefit: automation reduces pain, which makes you more likely to perform the task more often.

As a basic example, suppose you deploy manually to production, and on average it takes you about three hours. You currently deploy to production about once a month. If automating the deployment takes that down to 10 minutes, you're only saving 2 hours 50 minutes a month. If it will take you a couple of days to fully automate your deployments, it will take months to see a return on that investment. Is it worth it?

But there's more to the story. If your deployment now takes 10 minutes and a few clicks (rather than managing multiple remote desktop connections, etc.), and your deployments are reliable, you will be less reluctant to deploy to production. You might even start to deploy to production every week. Perhaps even multiple times a week. This means you can get features into the hands of users even sooner. By reducing the cycle time, you improve the feedback loop, and reduce your software inventory.

It's easy to look at an established project and say "automating this deployment would take longer than the time we'd save". But what is the opportunity cost of not automating?

Octopus Deploy has had two different ways to execute PowerShell scripts. Initially we hosted PowerShell inside an AppDomain, and invoked a pipeline. That approach had problems, so now we simply call PowerShell.exe over the script and pipe the results.

One problem I hit when I changed our approach is the way PowerShell uses return codes. I assumed (wrongly) that if a script ran successfully, 0 would be returned, and if it failed (an exception, invalid syntax, etc.) the code would be non-zero.

So here's a test - create this file in PowerShell:

# Call a command that doesn't exist
rubbish

Now run it, and print the return code it returns:

Now try this script:

# Parser error
!

Another:

# Invalid argument
New-Item -Dwarf Sneezy

Obviously, exit code 0 in PowerShell can signify anything from "the script ran perfectly" to "your script is so utterly broken that PowerShell will be uninstalled". One might argue that this is the correct behavior, because PowerShell did its job successfully (running a script) but that the script was incorrect, and that's not PowerShell's fault. Personally, I disagree with that approach. It would be like the C# compiler returning exit code 0 for code that doesn't compile, because it's not the compilers fault that the code is bad.

So, I couldn't rely on exit codes alone to tell Octopus that a script failed.

However, all of these kinds of errors do write to the stderr stream, which is good. So I figured, if the script is writing to stderr, there's some kind of problem with the script, and we can use a write to stderr as a signal to fail the script. That's what I did in Octopus 1.4.

But this didn't work so well in practice - it turns out people use stderr a lot for things that they don't want to fail the deployment. So I got a lot of bug reports about scripts that ran "OK" but were being treated as fails.

So now, I don't do anything with the return code (oh, except I do set the PowerShell exit code if $LastExitCode is non-zero). It's up to you to check for errors and return an appropriate exit code when your script runs. PowerShell makes it really easy to do that.

Writes to stderr will result in a warning in your deployment log, however. And you can use the special Octopus variable OctopusTreatWarningsAsErrors to have Octopus fail the deployment if there's a warning. So this is a way you can have Octopus fail when stderr is written even if PowerShell exited with code 0.

We're proud to be helping to sponsor the DevOps Down Under 2013 conference in Sydney.

DevOps Down Under is the southern hemisphere's premier DevOpsDays conference, this year running on the 12th & 13th of July in Ultimo, NSW, Australia. The conference provides a diverse range of talks, with a business level focus on the first day of the conference, and a technology focus on the second.

I'll be at the conference both days to hang out and engage with the DevOps community, and I love the open space format they are using for half the conference. They are still looking for session proposals, so if there's something you'd like to talk about now is a good time to suggest it. Hope to see you there!

Jeff French will be delivering a great presentation on the subject of Continuous Delivery of Complex Applications with TeamCity and Octopus Deploy:

One of the main tenets of Agile development is to deliver business value to the production environment early and often. That's easy enough if you are delivering one small web app, but what if your application is composed of several web apps across multiple tiers with a large database and maybe even a few Windows services and scheduled tasks? Now you need a deployment system that is built to scale and allows you to automate all of these tasks to achieve consistency in your deployments. In this talk I will show you how to deploy a complex application to multiple environments with just the click of a button using TeamCity and Octopus Deploy.

The first presentation will be at the Kansas City Developers Conference on May 4.

The second will be at the Norwegian Developers Conference on June 12.

Octopus Deploy uses RavenDB for data storage. Projects, releases, deployments, deployment logs, and so on are all kept in an embedded RavenDB database. The UI for Octopus is written in ASP.NET MVC, with most of the data rendering done using Razor.

These technology choices actually create a bit of a problem: RavenDB indexes are asynchronous, but we really want to render the most up-to-date information that we can when we render a page. RavenDB applications should be designed for eventual consistency, but standard ASP.NET MVC views/controllers lead you to assume you're always rendering the most up-to-date information.

We encountered this kind of bug report a lot when we first released the RavenDB version of Octopus:

I went to the "add machine" page, entered the machine information, clicked save, but when I went to the environments page, my machine wasn't listed. A few seconds later I hit refresh and there it was.

To fix these issues, I scattered this on almost all of the queries:

.Customize(x => x.WaitForNonStaleResultsAsOfNow())

In fact I did it so much that I created an extension method to make it easier:

.WaitForAges()

The upside of doing this is that every page always showed up to date information. The downside is that, once a server has too many documents, requests start timing out because the indexes are out of date. This is becoming a problem.

Designing for eventual consistency

Instead of the eventual consistency model being a weakness, for Octopus Deploy V2 I'd like to make it a strength.

When we render a page, we'll avoid doing any queries to RavenDB. Instead we'll render down the HTML/CSS/JS needed for the page. For each of the areas on the page, we'll use SignalR to fetch the data asynchronously. The result will include whether the index is stale, and if so, we'll let the user know ("this data may be out of date - last updated on XYZ").

Then, using SignalR, we'll subscribe to change notifications in RavenDB, and relay them back to the application to tell it that the data has changed.

Once an area of the page knows that there's new data available, it might:

• Show the updates immediately
• Show a message/refresh button

This approach should hopefully make the UI feel faster (since we can render information quickly without waiting for the indexes to be up to date) as well as feeling more real-time (no more hitting F5).