Introducing Damian Maclennan

The Octopus Deploy team has grown considerably in the last year, but because we've been a bit head down working on the product it's been a while since we've posted an update on the blog about who's who.

Over the next couple of weeks we're going to fix that and introduce you to the newer members of the team.

Damian Maclennan

My name is Damian Maclennan, I joined in September last year as CTO. If you read our newsletter last week then you've seen something by way of introduction to me. I've been close to Octopus as a product since its inception, having worked alongside Paul as a consultant with Readify. I spent nearly 6 years there helping teams be agile and ship better software through better agile practices, software design and automation.

Automation has always been a bit of a passion of mine, having championed build and deployment automation since the early 2000s in digital agencies, often a very uphill battle.

I'm at Octopus to do a number of things; grow the team and our development processes as well as get more people using Octopus by reaching out to people in the community to help them be successful.

My other technical interests are scalable distributed architectures, typically messaging based. I'm one of the instigators of a .NET messaging framework called Nimbus. I blog (all too infrequently like most people) and Tweet @damianm.

Growing Octopus Part IV: Interview process

This is part four in a series of posts about how we're growing as a company. In this post I'll talk about how we recruit, interview and onboard new team members.

  1. Developer career paths
  2. How we work
  3. Compensation and career progression
  4. Interview process
  5. Local, relocation, or remote?
  6. Everyone does support

Interview process

At the start of this series, I wrote:

Octopus grew by about 40% every quarter for the last 8 quarters, or 400% in the last 12 months. We hired 6 people over that time, so that leads to a conclusion: assuming we grow at the same rate this year, we're going to be hiring at least another 12 people.

Knowing that we'll be hiring heavily this year, one of our first challenges was to actually have a structured recruitment process. Our old process looked something like: we're hiring people we already know through our networks, so we'd have coffee with them, then make an offer. Clearly, that doesn't scale.

Damian spent some time designing a recruitment process which looks like this:

  1. CV's/resumes come in. We only use these to filter out people who definitely aren't worth talking to (wrong city, clearly not experienced, etc.), but otherwise they don't factor in very heavily.
  2. Phone interview: this usually lasts 20 minutes, and progresses through a handful of questions to give us a sense of whether it's worth progressing to the next stage (we err on the side of 'yes'). Think FizzBuzz-level of difficulty.
  3. In person interview

First up, don't worry: our process isn't a days-long process involving dozens of people. The in-person interview takes approximately two hours, and just involves just you, Damian and myself.

Part I: Pair programming exercise

First, there's a pair programming exercise. This takes between 45 mins to an hour. The goal is to help us determine:

  1. Can you actually write code: a little string manipulation, some very basic data structures, and some critical thinking. We don't expect you to be able to write the whole thing without a problem, and we'll help if you get stuck.
  2. Can you pick up new concepts. The problem domain we use in our exercise is something you've probably never heard of, but can be taught in 5 minutes. We'll spend some time at the start of the exercise making you a domain expert in the problem domain.
  3. Are you someone we'd actually want to pair with to work on real Octopus code each day. Are you personable, do you communicate when you get stuck, or do you get easily distracted or spend most of the session complaining about keybindings or your current employer.

We start with a stub solution in C#, but we're happy if you prefer to use a different language too. Someone will sit next to you as you do it, and they might ask questions about your approach or help you avoid coding into a corner. Once the basic version is working, we'll let you clean it up (we don't expect beautiful code at first, but we'd like you to refactor to something nice after), we'll add a few requirements, and get a sense of how you approach refactoring and object oriented structure.

Part II: Q&A

After the pairing exercise is a more traditional interview. We'll ask about your past experiences, and get a sense for what excites you - what are you passionate about? If you're applying for a senior developer/architect role, we also want to find something you can go really deep on - something that proves that you care about what goes on under the hood. We'll give you a choice of topics to choose your strength, such as:

  • Concurrency
  • Memory management/garbage collector
  • Production debugging/performance analysis
  • Architecting high-performance systems
  • HTTP/TCP fundamentals

For example, if you choose concurrency, we'll discuss processes vs. threads, thread synchronisation, pooling, what async/await actually does (your answer should probably describe IOCP, for example), and so on. As discussed in a previous post in this series, this is stuff we expect senior developers to know.

This is also a chance for you to ask us questions, since interviews are just as much about us interviewing you, as you interviewing us.

Part III: Presentation

Finally, for senior developer roles and above, we'll ask you to give a presentation for about 20 minutes. Typically this involves pitching a new feature for Octopus to us, or teaching us about something you're passionate about (development related, of course). The point of this is to get a sense of whether you can communicate ideas to the team, and structure your thought process to bring people along with you.

Decisions and feedback

We try to give feedback as much as we can throughout the process - e.g., if the pairing code is messy, we'll point out why we think so and give you a chance to tidy it. If you're struggling on a deep dive question, we'll fill in a gap or even move to a different topic.

At the end, if we're pretty sure the answer is no, we'll give you the answer on the spot - there's no sense letting you go home wondering, if our minds are made up. Either way, we'll let you know the answer within a couple of days, and we are happy to answer any questions about what went wrong/what you might have done differently - in fact we think giving honest answers to these questions is just as important as the initial interview. Brisbane is a small town, and we hope you'll come back in 6 months and try again.

The whole process takes about two hours, and we keep it casual and friendly. We've put half a dozen people through this process now, and I think it's working quite well - certainly better than our previous recruitment approach. I especially like the pair programming exercise as a way of getting to know what people are really like to work with. We all have implicit biases and reach conclusions in a matter of seconds when the interview starts, but seeing the person actual put down code can really undo those assumptions (for good or bad).

If you think you'd ace the interview (and live in Brisbane), we're hiring!

February Community Roundup

One of the really cool things about having a product like Octopus Deploy is when customers start to write their own blog posts about it. We're going to start putting together a list of those posts so that you can see what other people are doing around the world.

This is a roundup of what happened in February (a few days late I know).

IANA Assignment

Firstly though, we had a "wow, that's cool" moment recently when IANA officially recognized our tentacle communications port (10933).

In the community

We're not the only ones that have been busy lately. Some of our awesome customers have been writing blog posts about their success with Octopus Deploy.

We have to give a big shout out to Karoline Klever, who has written a series of posts about introducing Octopus Deploy to her organisation and all their customers. Sounds like a great journey! Also on the topic of culture change, Martyn Frank has blogged about his experiences with team culture with the introduction of Octopus and automation.

Our friend Ian Paulin has continued his postings on the changes in Octopus with a great summary of what's happening in 3.0.

Gregor Suttie also has started giving Octopus Deploy a Test Drive, we're looking forward to some more posts there.

Finally David Roberts has released a Google Chrome extension called OctoPygmy to customise the interface and do some filtering and grouping of environments and the dashboard. Looks very cool!

So that's the February roundup, look out for the next one in a few weeks!

Why Octopus uses self-signed certificates

Communications between the Octopus Deploy server and Tentacles are secured using X.509 certificates. When Octopus is installed, a new self-signed certificate is generated. When Tentacles are installed, they also generate a new, self-signed certificate. In this post I'd like to discuss why we use self-signed certificates - after all, isn't that bad?

Background

When the Tentacle is registered in Octopus, the thumbprints (which identify the public key of the certificates) are exchanged - that is, you manually paste the Octopus thumbprint into the Tentacle setup wizard, and you manually verify the Tentacle's thumbprint in the Octopus UI. This screenshot is from Octopus 1.0 but the same concept exists in 2.6:

Exchanging thumbprints

When Octopus connects to a Tentacle, it verifies that the Tentacle presents a certificate with the thumbprint it expects. Likewise, when Tentacle handles a connection, it verifies that the connection is from an Octopus server that it trusts. Both sides abort the connection if the other side isn't who they think it is.

Why self signed?

A question sometimes comes up around why we use self-signed certificates - isn't that a bad practice? To answer this, let's reverse the question: why use a certificate authority?

Suppose you want to do some online banking. You go to your bank's website, and are redirected to their HTTPS site.

Bank of America certificate

You don't actually know if this is Bank of America or not. But your browser trusts Symantec (who own VeriSign), and Symantec tells you "When someone ordered a certificate for BankOfAmerica.com, we phoned them and did some other checks, and we're pretty sure it's really Bank of America. And you trust us because we're Norton, right?"

Here's a more secure alternative to relying on certificate authorities (CA's) like Symantec: when you were in the bank branch setting up your account, shaking hands with the bank manager, they gave you a thumbprint for their website certificate. You write it down on some paper, and when you get home, you plug it in to your computer. You then wouldn't need any certificate authorities: your computer would trust BankOfAmerica.com only, without the need for any third parties.

We have certificate authorities for convenience, because expecting people to write down public keys for every site they plan to visit is too much effort. We don't have certificate authorities for security: they actually reduce the security. CA's get compromised: remember DigiNotar?

If Octopus used CA's

If CA's were used with Octopus, there are two ways we could use them:

  1. We just trust anything issued by a CA. Instead of checking whether the Octopus/Tentacle was trusted, we'd just assume that if they present a certificate issued by YourCompanyCA, then it must be OK to communicate.
  2. We verify the thumbprints anyway and use those to establish trust.

Approach #1 is dangerous because if the CA is compromised, then every machine on the network is effectively compromised, since anyone with a valid certificate could pretend to be anyone else. This applies whether the CA is internal to your organization, or a trusted third party. With our current approach, since we verify certificates individually, it's not possible for machine A to pretend to be machine B without having machine B's private key.

Approach #2 is better, except we're effectively ignoring the CA anyway. What value does the CA provide in this case if we are verifying thumbprints ourselves?

The argument could be made that Octopus should support use of CA's to issue certificates just for the 'feel good' factor as we've been conditioned to think that self-signed certificates are somehow intrinsically bad. You can use Tentacle.exe import-certificate to make Tentacles use a CA-issued certificate (but there's no way to do this on the Octopus Server currently), but I think the argument is driven by company policy ("Self-signed certs are forbidden!") rather than by sound reasoning.

The only argument we can think of as to why a CA would make sense from a security point of view in Octopus is for revocation lists. But you can always tell Octopus/Tentacle to no longer trust a given certificate without needing CRL's, so I think that argument falls down too.

We take security seriously in Octopus, and we also try our best to make it easy to use. We think the exchanging of thumbprints and use of self-signed certificates is the best solution from a security point of view, and adding any extra layers of CA support seems to make the software more complicated than necessary, and either adds no value, or creates an attack vector depending on how it would be implemented. I hope this post sheds some light on our reasoning.

Thanks to OJ from Beyond Binary for reviewing a draft of this post

Growing Octopus Part III: Compensation and career progression

This is part three in a series of posts about how we're growing as a company. In this post I'll talk about our plans for compensation and career progression.

  1. Developer career paths
  2. How we work
  3. Compensation and career progression
  4. Interview process
  5. Local, relocation, or remote?
  6. Everyone does support

Salary expectations and fairness

At places I've worked in the past, pay discussions typically start with "how much do you want?", and then negotiated downwards to how much you'll accept. Good negotiators get more, bad negotiators get less. Women and minorities tend to be at a disadvantage in this kind of system. Two people doing the same job, equally well, can often be getting paid very different amounts.

My goal is to have a fair system, whereby any two people doing the same role, with the same kind of experience, should be earning the same. Simon Galbraith helped define the acceptance criteria for this: if I accidentally printed a spreadsheet of everyone's salaries and left it on my desk, no one should be outraged to see what anyone else was earning. A few people have written about such structures, including at Fog Creek. We're still starting out (our first full time employee hasn't been with us for even 12 months yet) but here are my thoughts so far.

In my post about Developer Career Paths I outlined the different developer roles I imagine us having down the track. So our goal is that any two people in the same role (e.g., senior developer) should be on the same salary. To achieve this fairness, this means that the level of compensation for a given role is based on:

  • The role, and the market rate for that role (based on location). We'd always aim to meet the market.
  • Level of experience in the role - some people might only just make the cut for a given role, while others could be very close to moving into the next role

Pay rises work similarly; each year in July we'll review the market rates for a the different kinds of roles, adjust for inflation, and assess where people fit in. We'll then adjust across the board.

(Of course one downside of this approach is we can't advertise that a given role pays $X, since you'll then know how much everyone here is earning :-))

Coaching over performance reviews

My second goal is to decouple performance reviews from salary negotiations. We hire smart, professional people, and we expect them to perform well simply because doing a good job - building great software, making customers happy - is an essential part of being human. My father-in-law was a trained carpenter, and when he builds a chair or a table, he does his best not because someone else is watching him, but because he takes pride in what he creates. Intrinsic motivation is incredibly powerful.

Rather than quarterly/yearly performance reviews, we do regular one-on-ones. It's not a review, it's a coaching and feedback exercise: what can we do help you to be more autonomous? To contribute more?

Could all this mean that we miss out on good people because they can negotiate for slightly more elsewhere? Perhaps. In Brisbane, a large chunk of the software developer market is in contracting to government, so it's certainly possible to earn very high rates that we can't compete with (of course, I think we can compete by offering much more enjoyable work). But overall, my goal is to build a rockstar team, not hire a bunch of individual rockstars, so ensuring fair pay across the team is an important part of making that work.

Deployment targets in Octopus 3.0 (RFC)

In this post, I'd like to discuss some of the changes we're making to Tentacle, and deployment targets in general as part of Octopus 3.0.

Summary:

  • Multiple types of "machines" (Listening and Polling Tentacles, SSH, Azure Websites, Azure Cloud Services, WebDeploy, offline drops)
  • Tentacle as a shell
  • Open sourcing Tentacle
  • FTP deployments no longer supported

Multiple types of machines (deployment targets)

Octopus and Tentacle were designed together, and it was assumed that the only kind of "target" that Octopus would deploy to would always run Tentacle. In 2013, we added support for Azure Cloud Services and FTP sites, both to handle situations where installing Tentacle wasn't possible. These didn't appear on the environments page, weren't involved in health checks, and deploying to multiple of them in parallel (e.g., uploading to multiple FTP repositories) wasn't possible.

Some of our most common feature requests were to add support for new kinds of deployment targets: specifically, support for deploying to Linux servers, better support for deploying to Azure websites, and the ability to deploy to "offline" machines by creating a package that could be manually executed.

To do this, we'll be expanding the concept of "machines" in Octopus to a more general concept of "deployment targets". From the environments page, instead of adding a machine (which always meant a Tentacle), you can add different deployment targets:

New deployment target dialog for Octopus 3.0

Previously, for FTP and Azure cloud service steps to work, you had to have a special Tentacle running on the Octopus server. For 3.0, this will no longer be required; Octopus will be able to push Azure Websites/Cloud Services natively.

As you can see in the image above, we're adding a few new deployment targets. The SSH target allows you to deploy to any server that runs SSH, using either username/password authentication or certificates. These could be Amazon AWS Linux instances, your own on-premises Linux servers, machines running OSX, or even a Raspberry Pi. Instead of running PowerShell scripts, we'll execute bash scripts. You can read more about Linux/SSH plans in our RFC.

The WebDeploy target will allow you to deploy to any server that has WebDeploy enabled. Some third party hosting vendors, as well as Microsoft Azure, support WebDeploy. A specific Azure Website deployment target uses the WebDeploy client under the hood, but will be easier to set up (once you connect Octopus to your Azure subscriptions, you'll be able to pick the websites from a list).

Offline drops are a special deployment target which will bundle all the files needed to execute the deployment, and put them on a file share ready to be coped to a USB drive and deployed manually.

Just like Tentacles, these machine types can be tagged with roles, and you'll be able to deploy to them in parallel. For example, if you have two Azure websites in different regions, you could give them the same role, and then set that role as the target of a NuGet package step - the package will then be pushed to both sites in parallel.

Depending on the targets, deployment logic will be executed slightly differently. For Tentacles and SSH deployments, we'll execute config transforms and run scripts on the remote target machine. For Azure websites, cloud services and WebDeploy, they'll be executed on the Octopus server and then uploaded. For offline drops, they won't be executed at all, instead the code needed to execute them will be included in the bundled package.

Going forward, we have plans for more target types, specifically a PowerShell Remoting target. These are just the ones that will ship as part of Octopus 3.0.

Tentacle as a shell

Another big change we're making is to how Tentacle is currently designed. At the moment, Tentacle is an agent that communicates with Octopus. It knows how to do many things:

Tentacle currently

As an example, the C# code that makes up Tentacle knows how to do configuration transforms, modify IIS, and a load of other things. If we ever need to change how these features work, we we had to release a new version of Tentacle.

As we added SSH support, we realised that the same architecture couldn't work - all SSH can do is to run commands and move files. All the logic and conventions for configuration transforms, etc. would need to be pushed over from the Octopus Server.

Decoupling the communication logic (the SSH side of things) from the deployment logic is a much nicer architecture. So we've decided to do the same for Tentacles in 3.0: Tentacle will just know how to run commands or transfer files, over a secure connection. All the logic and tools for deployments will be sent up from the Octopus server, meaning Tentacle upgrades will be far less common.

Tentacle in future

From 3.0 onwards, think of Tentacle as a Windows-oriented version of SSH, except with support for both polling and listening modes. All it knows how to do is to execute PowerShell scripts, or transfer files. Everything else - the config transform stuff, the IIS stuff - will be PowerShell scripts that Octopus will ensure exist on the Octopus server prior to deployments executing.

Open sourcing the core package deployment components of Tentacle

Now that we're decoupling Tentacle the communication channel from Tentacle the deployment engine, we gain a new possibility: all the scripts and deployment logic that Tentacle executes during a deployment (as well as the scripts and tools used in SSH deployments) can now be open sourced. Our plan is to manage this as a GitHub project. If you don't like the scripts that we call during a deployment, you'll be able to fork this, build your own Tentacle package, put it on the Octopus server and use your own scripts/tools during the deployment instead. It will also mean we can take community contributions, which will be especially helpful for supporting the various Linux distributions.

Dropping support for FTP

Looking at our usage statistics, FTP is the least commonly used deployment target (~253 FTP steps in the world, compared to 36,000 NuGet package steps), and I suspect most of these are for Azure websites.

Since we're adding native WebDeploy and Azure Website support, it's hard to justify continuing to maintain FTP as a standalone step or machine type. For this reason, we'll be deprecating it. We'll make a step template that uses a command-line FTP tool which could be used as a replacement instead.

Conclusion

We still plan to release a preview of Octopus 3.0 around late March/early April, even if all these deployment targets aren't fully complete by then. In the mean time, we'd really appreciate your comments on our plans above, especially on open sourcing the core deployment scripts/tooling. Let us know in the comments below!

How we create the Tentacle Army

One of the challenges of being an independent software vendor is using your software on infrastructure that resembles what your clients are using. Octopus Deploy gets installed on a variety of operating systems and deploys over every network configuration imaginable to targets ranging from a single tentacle to a thousand tentacle monster. Our typical development configuration consists of a server and one or two tentacles residing on the same machine. In that configuration everything works fine, it isn't until Octopus gets deployed into the wild that it starts to unravel.

The Tentacle Army

The Tentacle Army is our attempt to replicate the most demanding configuration that Octopus will run in. Before a release of Octopus is published we fire up the Tentacle Army and attempt to break the release. At the moment the Tentacle Army consists of 800 tentacles located in Singapore and Brazil, regions chosen for their high region-to-region latency. In this configuration we have 40 tentacle virtual machines with 20 tentacle instances per machine.

Provisioning the Tentacle Army

We use Amazon AWS to provide the infrastructure for the Tentacle Army. A configuration file specifies what we want in our test environment:

{
  "OctopusServerConfiguration" : {
      "Environments" : [ "Development", "Test", "Production" ],
        "InstanceType" : {
            "Value" : "m3.medium"
          },
      "NumberOfInstances" : 1,
      "Platform" : "OctopusServer",
      "PrimaryStorage" : 40,
      "Projects" : [ {
            "Name" : "Multi release project",
            "DeploymentSteps" : [ {
                  "Name" : "Deploy package",
                  "PackageId" : "100mb",
                  "PackageUrl" : "https://s3-ap-southeast-1.amazonaws.com/octopus-testing/100mb.1.0.0.nupkg.zip"
                } ],            
            "Releases" : [ {
                  "Environment" : "Development",
                  "RepeatAfterMinutes" : 10,
                  "RunAfterMinutes" : 5
                } ]
          }],
      "RegionEndpoint" : "ap-southeast-1",
      "ServerInstallerDownloadPath" : "http://download.octopusdeploy.com/octopus/Octopus.2.5.12.666-x64.msi",
      "ToolsInstallerDownloadPath" : "http://download.octopusdeploy.com/octopus-tools/2.5.10.39/OctopusTools.2.5.10.39.zip"
    },
  "TentacleConfiguration" : [ {
        "InstanceType" : {
            "Value" : "m3.medium"
          },
        "Mode" : "Listening",
        "NumberOfInstances" : 5,
        "NumberOfTentaclesPerInstance" : 5,
        "Platform" : "Server2012_R2",
        "PrimaryStorage" : 40,
        "RegionEndpoint" : "ap-southeast-1",
        "TentacleInstallerDownloadPath" : "http://octopusdeploy.com/downloads/latest/OctopusTentacle64"
      } ],
  "VPCConfiguration" : [ { 
        "RegionEndpoint" : "ap-southeast-1"
      } ]
}

Through the configuration file we can specify:

  • the platform (we test Server 2008 32, 2008 64, 2012, 2012 R2)
  • the version of the Octopus server and tentacles
  • the number of tentacles and their location
  • environments, projects and automatic deployments
  • domain controller configuration

The configuration is fed into an application that calls the AWS API to provision the required machines. Once the machines have been provisioned they are set up via PowerShell. In addition to the Octopus server and tentacles, we also provision a NAT and, if required, a region-to-region VPN. The Tentacle Army ends up looking something like this:

Tentacle Army diagram

To infinity and beyond

The Tentacle Army has helped us to better relate to customers who put Octopus under pressure. We have been able to identify areas where Octopus struggles and felt the experience of managing a large number of tentacles through our UI. With a bit more work we will be collecting metrics so that we can benchmark our releases and validate the improvements that we continue to make on Octopus. We will also provide Tentacle Army as a service, allowing any team member to fire up a test environment and integrate the Tentacle Army into our automated process.

Growing Octopus Part II: How we work

This is part two in a series of posts about how we're growing as a company. In this post I'll talk about how we collaborate as a team.

  1. Developer career paths
  2. How we work
  3. Compensation and career progression
  4. Interview process
  5. Local, relocation, or remote?
  6. Everyone does support

Where we work

Entrance

When Octopus was just me, I worked from home. As we grew, we continued to work from home, chatting online or over the phone. Developing software requires collaboration, so since we were in the same city, we started to meet up on a regular basis in public places like cafés and libraries. For about a year, our regular meeting place was the Queensland State Library. Meeting once a week in person was a great way to have conversations that were harder to have online.

At first we just worked from the state library café, but as we grew to five or six people finding a café table (and power) became exponentially harder. We started to book meeting rooms at the library, but they were sometimes hard to get, and had a 4 hour per-day limit which meant we'd have to go home after lunch.

As a team we decided that continuing to meet in person was important, but the library and public spaces weren't working. We had also tried the iLab, who were kind enough to give us some desk space, but since we weren't using it each day we felt like we were imposing on other people's space - at least public space belongs to everyone. So we decided to get an office.

It took a few months to find the right place, but we eventually settled at 4/540 Queen Street. It's now turned into - in my opinion - a great little place to work:

Entry

Work area

View outside

View more photos

We treat the office similar to how we used the library: it's a place to come when we need to collaborate, even if we still predominantly work from home. We meet as a team every Wednesday (more on that below), and then we might come in on Friday for interviews, or on Tuesday to do some pair programming/designing for an upcoming feature. The office is a place to use if you need to use it. If you don't, avoid the traffic and stay home. On days when I do find myself in peak hour traffic, I'm reminded just how much more time I gain with my family most days of the week.

Schedule

We hire professionals, so we trust them to work and manage their time, and judge based on contribution rather than on time spent at desks. We do a daily standup each day around 9:30AM via Google Hangouts (or in person if in the office). Aside from that, there are no core hours or key times, and we trust people to work whichever way suits them.

Standups via Hangouts

Aside from the morning hangout (and support calls, which I'll discuss in a future post), we communicate pretty asynchronously throughout the day using Slack. One thing I'm proud of is that we use email extremely rarely internally (customer communication involves a lot of email, unfortunately). If you need to talk to a team member, either chat on Slack, ask them to jump on Hangouts (a permanent Hangout is always ready to use), or Skype the person you're trying to reach. It's rare to check your email in the morning and see an email from a team member.

Every Wednesday is our meet up face to face day, when we all come to the office. What we do that day will vary, but having a fixed day works well since you can plan for it and save questions/conversations for that day.

We do two week sprints involving the whole company, which end and start every second Wednesday. Sprint start/end Wednesday typically looks like this:

  • Sprint review: everyone does a short presentation on what they've been up to during the sprint. Developers working on a feature together might do a demo. Vanessa does a presentation summarising the key support tickets and trends. We cover hiring decisions, office improvements, marketing activities - basically anything that we did over the previous two weeks. It's great to see people share what they've been up to, and it's the highlight of my fortnight.
  • Retrospective: what worked? What didn't work? What can we try differently this week?
  • Sprint plan: what are we going to do this sprint?

We use GitHub issues to track what we're working on, using a public issue repository for public things, as well as a private one. Huboard gives us a task board view over both issue lists combined. Since we have "One Sprint", this means that the backlog contains a mix of tasks: development, operations, marketing, support, documentation, office renovation work; anything anyone is spending time on this week. We discuss what's involved with any items in the sprint, and we decide whether some items need any research or a short spec/RFC written to help figure out how it will be implemented.

On the "off-Wednesday" (in the middle of a sprint) we use the day to pair up, discuss features, or to do a bug bash or documentation bash. If you haven't tried a bug bash, they are great fun and have been the single best way we've found of finding issues that can't be discovered via TDD (and are way better than a dedicated tester).

As far as process goes, that's about it. Daily standups via Hangouts, weekly meetups in person, work from home or in the office - it's up to you. Sounds like a place you'd like to work? We're hiring!

Growing Octopus Part I: Developer career paths

Octopus grew by about 40% every quarter for the last 8 quarters, or 400% in the last 12 months. We hired 6 people over that time, so that leads to a conclusion: assuming we grow at the same rate this year, we're going to be hiring at least another 12 people if not more. And while Octopus Deploy is our only product now, long term our goals are much more aspirational: I'd like to see us become the next JetBrains or Atlassian, building a whole suite of world-class products that do one thing well, and which developers love to use.

This is part one in a series of posts about how we're dealing with some of that growth:

  1. Developer career paths
  2. How we work
  3. Compensation and career progression
  4. Interview process
  5. Local, relocation, or remote?
  6. Everyone does support

Developer career paths

Developer career paths

Building software like Octopus Deploy is incredibly fun, but also quite technically challenging. It's certainly very different to building enterprise software or typical web applications. Octopus is shrink-wrap software that needs to work (and be supportable) across a huge number of OS and network configurations. Most of the code is actually in the back-end, in orchestrating deployments across, in some cases, up to a thousand individual servers. There's a custom communication stack, there's language parsing and mini-programming languages, there's a huge amount of integration with cloud and with OS components. And when things go wrong, it's on a client's server that you have no access to, so you'd better have good "spidey senses" for debugging and diagnosing issues.

We need to grow this team, but we are not looking for 10x developers" or "rockstars". I'm much more interested in building an incredibly productive team than in finding great individual contributors. The kinds of skills we need might be more at home in a database vendor or infrastructure provider, not enterprise software development, but I think with the right attitudes these skills can be taught.

Given the size of our team in the next 12 months, I think that a career path for developers at Octopus Deploy - and the roles we'll be hiring for - looks something like this:

Junior developer

Given some mockups, and good guidance, should be able to implement small features. They should be able to come up with at least one solution to the given problem, but it may not be the most elegant, and it will probably be slow. They might not fully understand the tradeoffs they're making, and will occasionally get stuck. The most important attributes we're looking for in this category is eagerness to learn, and the ability to put your hand up and ask for help when you do get stuck.

Developer

Should be able to implement bigger features end-to-end, without so much guidance. Should be able to contribute to the design of features (both architecturally and from a user-experience/functionality point of view). For a given problem, should be able to think of a few different approaches, and have a good understanding of the tradeoffs. Writing tests, refactoring, and using patterns and good engineering principles should be second nature. Knowledge of advanced topics like threading are not expected to be incredibly deep, but should extend past "I've played with async/await and BackgroundWorker". Should understand topics slightly outside of development, like basic networking. The attributes we're looking for here are a passion for continually sharpening the saw, and a focus on delivering high-quality, working software.

Senior developer

Junior developers get into trouble. Senior developers get them out of trouble. Should be able to design and implement entire feature areas end-to-end. Should be able to think of many solutions to a given problem, have seen many of them before, and be able to understand the tradeoffs. Beyond writing great code, should be able to coach and mentor other team members on the same. Should be able to demonstrate a deep knowledge of topics like threading, memory management, networking, and the underlying operating systems. Given a crash dump, should have some idea of what to do with it. Attributes we're looking for here are obviously a certain level of competence and leadership, but also a passion for helping to coach and improve the rest of the team.

Solution architect

I don't really like the "architect" title, and we certainly don't want ivory tower architects, but it's all I can come up with beyond "Even more senior developer". A solution architect should be a great communicator, able to bring the rest of the team along with them in designing and building software. They should have enough experience to understand a wide variety of approaches to systems design. Given a mission like "figure out how to make a hosted version of Octopus" or "create a new product called Octopus Build", should be able to put the entire systems together and come up with end-to-end solutions. Finally, solution architects should be able to work closely with our end customers, helping them to understand and adopt our software and adopt it in their organizations.

Beyond software architect, I'm not sure what comes next (mainly because we aren't hiring at that level just yet). As we grow, having a career path for developers (that doesn't eventually lead to management) is important, and I expect we'll add more specialization and levels to the above.

One of the bits of feedback I've heard is that these expectations are a little too high, and that we shouldn't expect senior developers to understand threading, or how the garbage collector works. What do you think?

At the moment, our focus is on finding developers and senior developers. I'd love to take on junior developers, but right now there aren't enough people to help mentor them, so that will have to wait until we have a couple more experienced developers in place. If you think you'd fit one of these roles (and live in Brisbane), we're hiring!

RFC - Cloud and Infrastructure Automation Support

Octopus has a number of customers doing infrastructure automation in cloud IaaS environments such as Amazon EC2 and Azure. We know a lot of customers are using Octopus Deploy as a central component to this, but we also know from talking to people that there are a few areas where this causes a bit of friction. We've got some ideas for things we can add to the product that will make this easier, but it is a huge area and we know everybody is doing it slightly differently, so we'd like your feedback.

What we know is that there's no way we can provide an easy interface that will suit the majority of people, there are simply too many variables and different patterns. We also know that the customers who want to go down this road are a pretty cluey bunch and not afraid of getting their hands dirty when it comes to writing some scripts to manage machines.

Finally what we know is that customers would like to use Octopus Deploy to do more of the orchestration of this process, and that's what we want to solve.

Two schools of thought

We see two distinct patterns when it comes to infrastructure automation, and it's about whether the environment is changed as part of a deployment or more loosely coupled.

Scenario 1 - Tie the environment to the deployment.

A new release is ready to go live. Brand new instances are created and configured, tentacles are installed and registered with the Octopus server. The release is deployed to the new machines, the new machines are put in the load balanced pool of servers and the instances running the old release are removed from the pool, de-registered with Octopus and destroyed.

The chief problem with this currently is that Octopus evaluates the deployment target machines at the beginning of the deployment. So if you have a PowerShell step in your deployment that creates new machines, they wouldn't be part of the deployment. Because of this, the orchestration of the deployment needs to take place outside of Octopus. Many people use their build server for this, to run the script to create the environment then invoke the deployment via the Octopus API. This works but it is somewhat counter intuitive about where the control over deployment lives.

Scenario 2 - Manage environment separately

The alternate approach is that an environment is defined and managed separately. This is often the case for very elastic environments where machines can be added or removed automatically according to demand.

As an example, an AWS CloudFormation template is stored in source control. This file defines the shape of the environment and its behaviours and policies around automatic scaling. When the configuration is changed, scripts are run to push the new configuration to AWS causing EC2 to add, remove or change machines as needed.

While this works for people today, it requires extra scripts to be deployed and run for new instances to request any deployments. Additionally, as instances are destroyed Octopus will continue to poll them to carry out health checks so scheduled tasks are created to de-register deprovisioned machines.

Enter Environment Policies

A solution to the problems in Scenario #2 would be to have some behaviour rules on a per environment basis.

Dynamic Registration

Allowing dynamic registration for an environment would allow machines to be de-registered and "forgotten" if they don't respond to health checks or can't be contacted at deployment time. If a machine in a dynamic environment isn't responding, Octopus will remove it from the environment and stop trying to reach it. Octopus could keep a list of "Recently Removed" machines on the environment page as a visual indication.

Automatic Deployments

Enabling Automatic Deployments for an environment would initiate a new deployment whenever a machine came online. If a new machine instance is created as the result of an auto-scaling or machine recycling policy, when the tentacle is registered Octopus would evaluate what releases it should have and create a deployment to it. This would also be useful in environments that are intermittently connected and can miss releases (i.e. we have some customers who need to deploy to client machines with VPNs that are often offline).

Re-evaulate Machines Step

A simple solution to scenario #1 would be for Octopus to have the ability to re-evaluate the deployment targets midway through a deployment. A PowerShell script which creates new machines could set the new instance names in a variable which could be used to re-target the deployment.

Direct support for IaaS platforms

The features outlined above are some basic pieces of functionality which will enable more scenarios for lots of our customers. For deeper hooks into the various IaaS platforms we think that we'll be best served by releasing Step Templates in our library and potentially some open source scripts, tools and sample projects to help you best tailor an Octopus Deploy project to your needs.

We need your feedback

If you are doing infrastructure automation, would these options make your life and / or scripts easier ? If you're wondering how to do it, would this help ? If you're doing something that isn't helped by these ideas at all, what are we missing ?