March 2015 Community Roundup

There has been so much goodness from our friends out there in the community this month.

First up is Gregor Suttie blogging about "One Week using Octopus Deploy" and his need for Blue/Green Deployments.

David Roberts snuck this gem in right at the end of Feb: OctoPygmy - a Chrome Extension customizing the dashboard to filter parts of the Octopus Deploy UI by Project Groups, Environments or Roles. We think it's a cool idea and if you're managing a large environment it's worth a look.

Over on the OPTEN blog, Tim Pickin has been running a series on their Octopus Odyssey, automating the deployment of a complex multi-tenant web application. Some good lessons and ideas in there.

You might remember Karoline Klever and her blog series from the last newsletter. She's done it again with the Octopus Deploy Lab, a set of exercises developed for a workshop to help get Octopus neophytes up and running.

Are you in London? If so, get along to the Zimbra Social Usergroup on the 1st of April to find out from Rich Else about how the Macmillan Cancer Support Community used agile, continuous delivery and Octopus Deploy to upgrade their online community platform.

Also while we're in London, Ed Anderson recently did a good post on the state of .NET web app cloud deployments.

Next up our friend Jason at Domain, recently used Octopus Deploy to help with a rather tricky AWS migration. MacGyver would be proud.

Last but not least, if you're in Chicago in April, get along to the Chicago Code Camp on the 18th. It looks like a fantastic program and Ian Paullin will be presenting an Intro to Octopus Deploy. So if you know anybody who needs that sort of knowledge, send them along.

Introducing Damian Brady

Damian Brady

My name is Damian Brady (Damo to avoid mixups with DamianM), and I joined Octopus Deploy very recently at the end of February 2015 as a Solution Architect. Prior to Octopus, I worked as a developer, business analyst, and trainer in a range of companies. Most recently, I spent 4 years being exposed to many varied teams and industries as a consultant at SSW. I cut a lot of code, but I also spent a lot of time introducing teams to better ways to work.

I became aware of Octopus less than a year ago, and it’s fair to say I was hooked straight away. It became a go-to tool for any development team I was involved with. I introduced it to clients, wrote internal tools that used its API, and spoke about it, repeatedly, at conferences. Joining the team at Octopus was a natural (albeit challenging) step.

I’m here primarily to help the development community get the most out of Octopus and improve their DevOps process. I’ve always loved pushing people to take that next step to get better software out faster, and this is a great opportunity to do that.

I’m also a Microsoft Visual Studio ALM MVP, so I’ll be spending a bit of time making sure there’s the best possible ALM story for Octopus and Microsoft devs.

Growing Octopus Part VII: Local or remote team members

This is part seven in a series of posts about how we're growing as a company. In this post I'll talk about some of the decisions we've struggled with when trying to decide whether to only hire locally, or to hire remote team members.

  1. Developer career paths
  2. How we work
  3. Compensation and career progression
  4. Interview process
  5. Laptop program
  6. Anti-harassment policy
  7. Local or remote?
  8. Everyone does support

Remote vs. REMOTE

When people talk about working remotely, one detail they often leave out is that they're usually still working with people in similar timezones, just not from the same office. Consider these two scenarios:

  • I live in Chicago and work remotely with a team in New York. Sometimes I schedule a call at 2:00 PM and they forget to turn up.
  • I live in Australia and work remotely with a team in New York. I was awake until 2:00 AM and they forgot to turn up.

There's a world of difference (literally!) between these two.

That was the case for me: I was in South Australia, working with teams in Victoria and NSW, and it worked just fine. Scott Hanselman has great advice for working remotely, but he's in a similar situation: he's in Portland, and most (I assume) of his team are in Seattle. Modern technology solves all the hurdles of remote work, unless the person on the other end of the call is asleep.

When I lived in London, and our team in London had to work with a team in Tokyo, it was a completely different story. When time zones don't overlap, it's too hard to have lots of conversations, so people are left out. Trust is very hard to build over distance; it's just too easy to assume that those people, who are impossible to contact and never around when you need them, are the villians, acting out of malice or incompetence (or both!), while you're the ones trying to do the right thing. Phil Haack gives an excellent explanation of this:

Everyone is the protagonist of their own narrative. And in this narrative, it's only natural to see ourselves as the proverbial "good guy" of the story. We tend to rationalize our own actions as necessary or positive, much like Walter White until (spoiler alert) the end of Breaking Bad.

When you're in the USA and considering hiring remotely, you get the whole population of the Americas before you really need to worry about timezones. Things are a bit different from Australia. There's pretty much no time that a person in Australia can talk to a person in New York between 9-5 in their local time zone, so someone will always need to be around at odd hours.

I liked working remotely, so I always planned to build a remote working company. But I lived in Brisbane. When it was time to bring on someone full time, it was just easier to find someone locally. Then we found another person in Brisbane, and another. When you bootstrap, those first hires are so critical; adding a remote aspect adds a whole new level of difficulty and risk.

Resistance is futile

Until now, we've concentrated our efforts on finding team members in Brisbane. Yet, Australia accounts for only about 15% of our customers; the vast majority are in the USA and UK. At some point, we'll need a US and UK presence to be able to provide deeper support to customers in those time zones. Eventually we're just going to have to do it.

Working with people in another timezone requires big changes to the way we work, and it's better to do this now, while we're small and it's easier to effect changes to our communication patterns before the way we currently work becomes more entrenched. We're already part-way there - most of the team work from home most of the time - but to work across timezones will be harder still.

We're hiring in the USA/Canada!

As of today, we're now looking for remote team members who live in USA or Canada, as well as in Australia. A few thoughts we have so far:

  1. We're limiting the countries so as to limit the legal work involved, and to make basic things like paying people easier. We might look for contractors initially, but at some point I imagine we'll need a legal US presence. Better if there's only two countries to deal with than 10.
  2. We'll fly new team members to Brisbane for a month to work alongside us. The hope is that this will help get them productive and build that initial understanding and empathy, before we add the remote aspect. We'll also bring them here once every 6 to 12 months to keep it up.
  3. We'll go big: instead of just one person, we'll probably look to bring on 3-4 in a short time period, just so that they can form their own team and be more autonomous (and have people to talk to during work hours).

Growing Octopus Part VI: Anti-harassment policy

This is part six in a series of posts about how we're growing as a company. In this post I'll talk about our anti-harassment policy.

  1. Developer career paths
  2. How we work
  3. Compensation and career progression
  4. Interview process
  5. Laptop program
  6. Anti-harassment policy
  7. Local or remote?
  8. Everyone does support

Anti-harassment policy

One of the nice things about a small company is that there aren't a lot of HR policies to follow. In fact, we really only have a leave policy (like Balsamiq's: take some), and this anti-harassment policy.

Our desire for an anti-harassment policy was driven by the excellent work by Ashe Dryden on why conferences need a code of conduct. She makes a great argument about why conferences need a code of conduct, even if there have been no problems in the past:

A Code of Conduct is a public statement that sets the ground rules for participating in an event. Our conferences boast attendees from all over the world, coming from many different backgrounds and experiences; our expectations of what is appropriate are not always in line with one another. A policy like this ensures that everyone is on the same page without any gray areas and even better, that there will be less rules-lawyering. The people most affected by harassing or assaulting behavior tend to be in the minority and are less likely to be visible. As high-profile members of our communities, setting the tone for the event up front is important. Having visible people of authority advocate for a safe space for them goes a long way.

If this applies at conferences, it should certainly apply in the workplace too. With that in mind, here's our anti-harassment policy:

Octopus Deploy is dedicated to providing a safe, harassment-free working environment for everyone.

We do not tolerate harassment in any form. To be clear, harassment includes but isn't limited to:

  • Physical violence, threats of physical harm, or verbal abuse
  • Offensive verbal comments about a person's gender, age, sexual orientation, disability, physical appearance, body size, race, or religion
  • Stalking, following, harassing photography or recording, inappropriate physical contact, and unwelcome sexual attention
  • Any other actions or words designed to intimidate, denigrate or insult specific people or a group of people
  • Sexual language and imagery is not appropriate for any workplace or work-related activity, including in the office, via chat, email, in training events, or at parties.

This policy applies to all employees, office holders, contractors, consultants, guests, partners, vendors and volunteers.

If a person engages in harassing behaviour, managers or directors may take any action they deem appropriate, including warning the offender or termination.

If you are being harassed, notice that someone else is being harassed, or have any other concerns, please contact Paul (my number), Sonia (her number), or your manager. We will be happy to help contact the police, record any complaint, arrange alternative work routines, or otherwise assist those experiencing harassment to feel safe.

Introducing Shane Gill

Damian Maclennan

My name is Shane Gill, I joined Octopus Deploy in October 2014 as a test engineer. My previous experience has been focused on helping small businesses ship software. I have worked in a range of industries, most recently creating Minecraft plugins.

I wanted to work at Octopus Deploy because I was inspired by the culture and values that ooze from everything Octopus. Organic growth, customer focus, transparency and work-life balance are cliches often used but rarely applied. Octopus delivers in all respects, the team embodies these values and the customer really is number one. The focus on being good to people is evident in the awesome product.

I was hired at Octopus to help reduce the number of releases per version. If you are a developer, you would be familiar with working your butt off, putting out a new version and then getting hammered by your users with a long list of bugs and faults that your development process did not pick up. Release after release of patches and fixes ensues (along with the stress of support and the feeling of letting your users down). My initial mission was to ease that pain with the Tentacle Army.

In addition to raising armies of tentacles I am the brewer of the infamous OctoAle. Tweet @tothegills for the secret recipe.

Growing Octopus Part V: Laptop program

This is part five in a series of posts about how we're growing as a company. In this post I'll talk about our laptop program, which lets new team members pick the hardware that works best for them. Rather than just describe the policy, I thought I'd just post the policy as-is.

  1. Developer career paths
  2. How we work
  3. Compensation and career progression
  4. Interview process
  5. Laptop program
  6. Anti-harassment policy
  7. Local or remote?
  8. Everyone does support

Laptop program

Macbook Pro Retina

Great knowledge workers need great tools. Since we vary between working from home and working from an office, it's important to be able to have a computer that is portable and can be taken with you.

It's all too common to meet professionals who invest in great technology at home, but are forced to work with sluggish machines at work. Rather than have the company buy a fleet of laptops to give to people, I wanted to find a way to balance three goals:

  1. You should be able to pick the laptop/device that works best for you
  2. You should own it, not the company
  3. You shouldn't be out of pocket - the company should pay for it

Picking your laptop

You are a professional, and you are smart, and we trust you. You're intrinsically motivated to do great work because you take pride and derive great satisfaction in what you do, and you know the hardware that fits the way you work best. Perhaps you love the sleek look of a Macbook Pro. Maybe you prefer a touch screen. We trust you to pick the right hardware for you, because we know you'll pick whatever enables you to do your best work.

That said, we do have a few requirements/limits:

  • Maximum budget: AUD$4400 not including GST
  • It needs to be fast and powerful enough to do the work you need to do (compiling code, etc.).
  • You need a warranty that will cover the laptop and any repairs it may need for at least two years (this needs to be included in the price)
  • It needs to be new, not second-hand

We prefer to buy directly from the manufacturer (Apple, Lenovo, etc.) so we'd suggest shopping at their online stores rather than going to the local computer store. Once you've configured the system to your exact needs, send a quote to me and Octopus will purchase it and arrange for it to be delivered to you.

How it works

Any individual can salary sacrifice a laptop by working with their employer - the idea is that the employer purchases the laptop on your behalf, taking it out of your pre-tax salary. It's then yours to own. But the downside is that since it comes out of your salary, you're out of pocket.

To avoid being out of pocket, our program works like this:

  1. We increase your salary by CostOfLaptop / 2. I.e., for a $4,400 laptop, that's $2,200 per year
  2. You choose the laptop you want, we purchase it on the Octopus Mastercard, and ship it to your address
  3. We treat it like a salary sacrifice, by deducting CostOfLaptop / 24 from each month's pay. Two years later it is repaid.

The end result is that you end up with a new laptop, but it never comes out of your agreed salary, because we increase your salary and then salary sacrifice it back. Octopus has paid for it, but you own it.

Tax and other issues

When we purchase the laptop, the GST is claimed by Octopus. The total value that you salary sacrifice (and that we increase your salary by) is the after-GST portion. The salary sacrifice portion is pre-tax.

The ATO allow individuals to salary sacrifice one laptop and/or phone per year, as long as its used for your employment (even if you also play World of Warcraft on it at nights). This article has more details on what can and can't be salary sacrificed. Note that this means that if you already salary sacrificed a laptop under your previous employer, you might not be able to participate in this program until the next tax year (you should talk to your accountant).

It goes without saying, but as with all tax related issues, you should talk to your accountant about any of this before proceeding.

Note that when it comes to depreciation, some accountants will give different advice. Some may say it's fine to claim depreciation in your tax return since you own the laptop; others may say that the ATO doesn't allow it and that it's double dipping (since you already got the laptop tax-free since the sacrificed amount is pre-tax). Whether you claim depreciation is totally a matter for you and your accountant to figure out (smile)

Replacement

You can replace the laptop every two years - effectively the laptop will have been "paid off" in that time, so you're free to either keep using it, or get a new one under the program.

Lost/stolen/damaged

The laptop belongs to you, so if it's stolen, defenestrated, or falls down a well, it's your responsibility to find a replacement. It's a good idea to call your home and contents insurance provider to ensure the laptop is covered.

Leaving

If you happen to leave (we hope you never leave!) before the two years have passed since purchasing the laptop, the remaining balance from the salary sacrifice will be deducted from the final pay slip. If the laptop is relatively new and in good condition, at our discretion we may agree to take the laptop in lieu of the final balance.

Example

To help put this into concrete terms, here's an example:

  1. Roger joins in January and uses the program to order a laptop, with an after-GST value of $4400 on 1st January
  2. His yearly salary increases by $2,200 ($183.33 per month)
  3. The laptop is purchased on his behalf by the company, and is delivered to his address in time for him to start
  4. $183.33 is salary sacrificed from his wage each month, to repay the laptop. His pay slip looks like this:
    Salary $X + 183.33
    Deduct laptop salary sacrifice: $183.33
    Gross pay: $X
  5. Roger leaves to start his own business; his last day is 1 July
  6. In the 6 pay runs between his start and end date, $183.33 x 6 has been paid back ($1099.98)
  7. In his final pay run, the final balance of $3300.02 is deducted from the final payout figure
  8. Roger keeps the laptop

Additional hardware budget

Our laptop program covers the purchase of the laptop, and any software, warranties or other items that you buy at the same time from the laptop vendor. However, you might need other items, either to help work from home or from the office. Examples are headsets, cameras, keyboards, and so on.

Everyone has a budget of $400 per financial year, per person, to spend on these items. You can either buy them yourself and then send an expense claim to us for reimbursement, or we can buy them for you.

Introducing Damian Maclennan

The Octopus Deploy team has grown considerably in the last year, but because we've been a bit head down working on the product it's been a while since we've posted an update on the blog about who's who.

Over the next couple of weeks we're going to fix that and introduce you to the newer members of the team.

Damian Maclennan

My name is Damian Maclennan, I joined in September last year as CTO. If you read our newsletter last week then you've seen something by way of introduction to me. I've been close to Octopus as a product since its inception, having worked alongside Paul as a consultant with Readify. I spent nearly 6 years there helping teams be agile and ship better software through better agile practices, software design and automation.

Automation has always been a bit of a passion of mine, having championed build and deployment automation since the early 2000s in digital agencies, often a very uphill battle.

I'm at Octopus to do a number of things; grow the team and our development processes as well as get more people using Octopus by reaching out to people in the community to help them be successful.

My other technical interests are scalable distributed architectures, typically messaging based. I'm one of the instigators of a .NET messaging framework called Nimbus. I blog (all too infrequently like most people) and Tweet @damianm.

February Community Roundup

One of the really cool things about having a product like Octopus Deploy is when customers start to write their own blog posts about it. We're going to start putting together a list of those posts so that you can see what other people are doing around the world.

This is a roundup of what happened in February (a few days late I know).

IANA Assignment

Firstly though, we had a "wow, that's cool" moment recently when IANA officially recognized our tentacle communications port (10933).

In the community

We're not the only ones that have been busy lately. Some of our awesome customers have been writing blog posts about their success with Octopus Deploy.

We have to give a big shout out to Karoline Klever, who has written a series of posts about introducing Octopus Deploy to her organisation and all their customers. Sounds like a great journey! Also on the topic of culture change, Martyn Frank has blogged about his experiences with team culture with the introduction of Octopus and automation.

Our friend Ian Paulin has continued his postings on the changes in Octopus with a great summary of what's happening in 3.0.

Gregor Suttie also has started giving Octopus Deploy a Test Drive, we're looking forward to some more posts there.

Finally David Roberts has released a Google Chrome extension called OctoPygmy to customise the interface and do some filtering and grouping of environments and the dashboard. Looks very cool!

So that's the February roundup, look out for the next one in a few weeks!

Growing Octopus Part IV: Interview process

This is part four in a series of posts about how we're growing as a company. In this post I'll talk about how we recruit, interview and onboard new team members.

  1. Developer career paths
  2. How we work
  3. Compensation and career progression
  4. Interview process
  5. Laptop program
  6. Anti-harassment policy
  7. Local or remote?
  8. Everyone does support

Interview process

At the start of this series, I wrote:

Octopus grew by about 40% every quarter for the last 8 quarters, or 400% in the last 12 months. We hired 6 people over that time, so that leads to a conclusion: assuming we grow at the same rate this year, we're going to be hiring at least another 12 people.

Knowing that we'll be hiring heavily this year, one of our first challenges was to actually have a structured recruitment process. Our old process looked something like: we're hiring people we already know through our networks, so we'd have coffee with them, then make an offer. Clearly, that doesn't scale.

Damian spent some time designing a recruitment process which looks like this:

  1. CV's/resumes come in. We only use these to filter out people who definitely aren't worth talking to (wrong city, clearly not experienced, etc.), but otherwise they don't factor in very heavily.
  2. Phone interview: this usually lasts 20 minutes, and progresses through a handful of questions to give us a sense of whether it's worth progressing to the next stage (we err on the side of 'yes'). Think FizzBuzz-level of difficulty.
  3. In person interview

First up, don't worry: our process isn't a days-long process involving dozens of people. The in-person interview takes approximately two hours, and just involves just you, Damian and myself.

Part I: Pair programming exercise

First, there's a pair programming exercise. This takes between 45 mins to an hour. The goal is to help us determine:

  1. Can you actually write code: a little string manipulation, some very basic data structures, and some critical thinking. We don't expect you to be able to write the whole thing without a problem, and we'll help if you get stuck.
  2. Can you pick up new concepts. The problem domain we use in our exercise is something you've probably never heard of, but can be taught in 5 minutes. We'll spend some time at the start of the exercise making you a domain expert in the problem domain.
  3. Are you someone we'd actually want to pair with to work on real Octopus code each day. Are you personable, do you communicate when you get stuck, or do you get easily distracted or spend most of the session complaining about keybindings or your current employer.

We start with a stub solution in C#, but we're happy if you prefer to use a different language too. Someone will sit next to you as you do it, and they might ask questions about your approach or help you avoid coding into a corner. Once the basic version is working, we'll let you clean it up (we don't expect beautiful code at first, but we'd like you to refactor to something nice after), we'll add a few requirements, and get a sense of how you approach refactoring and object oriented structure.

Part II: Q&A

After the pairing exercise is a more traditional interview. We'll ask about your past experiences, and get a sense for what excites you - what are you passionate about? If you're applying for a senior developer/architect role, we also want to find something you can go really deep on - something that proves that you care about what goes on under the hood. We'll give you a choice of topics to choose your strength, such as:

  • Concurrency
  • Memory management/garbage collector
  • Production debugging/performance analysis
  • Architecting high-performance systems
  • HTTP/TCP fundamentals

For example, if you choose concurrency, we'll discuss processes vs. threads, thread synchronisation, pooling, what async/await actually does (your answer should probably describe IOCP, for example), and so on. As discussed in a previous post in this series, this is stuff we expect senior developers to know.

This is also a chance for you to ask us questions, since interviews are just as much about us interviewing you, as you interviewing us.

Part III: Presentation

Finally, for senior developer roles and above, we'll ask you to give a presentation for about 20 minutes. Typically this involves pitching a new feature for Octopus to us, or teaching us about something you're passionate about (development related, of course). The point of this is to get a sense of whether you can communicate ideas to the team, and structure your thought process to bring people along with you.

Decisions and feedback

We try to give feedback as much as we can throughout the process - e.g., if the pairing code is messy, we'll point out why we think so and give you a chance to tidy it. If you're struggling on a deep dive question, we'll fill in a gap or even move to a different topic.

At the end, if we're pretty sure the answer is no, we'll give you the answer on the spot - there's no sense letting you go home wondering, if our minds are made up. Either way, we'll let you know the answer within a couple of days, and we are happy to answer any questions about what went wrong/what you might have done differently - in fact we think giving honest answers to these questions is just as important as the initial interview. Brisbane is a small town, and we hope you'll come back in 6 months and try again.

The whole process takes about two hours, and we keep it casual and friendly. We've put half a dozen people through this process now, and I think it's working quite well - certainly better than our previous recruitment approach. I especially like the pair programming exercise as a way of getting to know what people are really like to work with. We all have implicit biases and reach conclusions in a matter of seconds when the interview starts, but seeing the person actual put down code can really undo those assumptions (for good or bad).

If you think you'd ace the interview (and live in Brisbane), we're hiring!

Why Octopus uses self-signed certificates

Communications between the Octopus Deploy server and Tentacles are secured using X.509 certificates. When Octopus is installed, a new self-signed certificate is generated. When Tentacles are installed, they also generate a new, self-signed certificate. In this post I'd like to discuss why we use self-signed certificates - after all, isn't that bad?

Background

When the Tentacle is registered in Octopus, the thumbprints (which identify the public key of the certificates) are exchanged - that is, you manually paste the Octopus thumbprint into the Tentacle setup wizard, and you manually verify the Tentacle's thumbprint in the Octopus UI. This screenshot is from Octopus 1.0 but the same concept exists in 2.6:

Exchanging thumbprints

When Octopus connects to a Tentacle, it verifies that the Tentacle presents a certificate with the thumbprint it expects. Likewise, when Tentacle handles a connection, it verifies that the connection is from an Octopus server that it trusts. Both sides abort the connection if the other side isn't who they think it is.

Why self signed?

A question sometimes comes up around why we use self-signed certificates - isn't that a bad practice? To answer this, let's reverse the question: why use a certificate authority?

Suppose you want to do some online banking. You go to your bank's website, and are redirected to their HTTPS site.

Bank of America certificate

You don't actually know if this is Bank of America or not. But your browser trusts Symantec (who own VeriSign), and Symantec tells you "When someone ordered a certificate for BankOfAmerica.com, we phoned them and did some other checks, and we're pretty sure it's really Bank of America. And you trust us because we're Norton, right?"

Here's a more secure alternative to relying on certificate authorities (CA's) like Symantec: when you were in the bank branch setting up your account, shaking hands with the bank manager, they gave you a thumbprint for their website certificate. You write it down on some paper, and when you get home, you plug it in to your computer. You then wouldn't need any certificate authorities: your computer would trust BankOfAmerica.com only, without the need for any third parties.

We have certificate authorities for convenience, because expecting people to write down public keys for every site they plan to visit is too much effort. We don't have certificate authorities for security: they actually reduce the security. CA's get compromised: remember DigiNotar?

If Octopus used CA's

If CA's were used with Octopus, there are two ways we could use them:

  1. We just trust anything issued by a CA. Instead of checking whether the Octopus/Tentacle was trusted, we'd just assume that if they present a certificate issued by YourCompanyCA, then it must be OK to communicate.
  2. We verify the thumbprints anyway and use those to establish trust.

Approach #1 is dangerous because if the CA is compromised, then every machine on the network is effectively compromised, since anyone with a valid certificate could pretend to be anyone else. This applies whether the CA is internal to your organization, or a trusted third party. With our current approach, since we verify certificates individually, it's not possible for machine A to pretend to be machine B without having machine B's private key.

Approach #2 is better, except we're effectively ignoring the CA anyway. What value does the CA provide in this case if we are verifying thumbprints ourselves?

The argument could be made that Octopus should support use of CA's to issue certificates just for the 'feel good' factor as we've been conditioned to think that self-signed certificates are somehow intrinsically bad. You can use Tentacle.exe import-certificate to make Tentacles use a CA-issued certificate (but there's no way to do this on the Octopus Server currently), but I think the argument is driven by company policy ("Self-signed certs are forbidden!") rather than by sound reasoning.

The only argument we can think of as to why a CA would make sense from a security point of view in Octopus is for revocation lists. But you can always tell Octopus/Tentacle to no longer trust a given certificate without needing CRL's, so I think that argument falls down too.

We take security seriously in Octopus, and we also try our best to make it easy to use. We think the exchanging of thumbprints and use of self-signed certificates is the best solution from a security point of view, and adding any extra layers of CA support seems to make the software more complicated than necessary, and either adds no value, or creates an attack vector depending on how it would be implemented. I hope this post sheds some light on our reasoning.

Thanks to OJ from Beyond Binary for reviewing a draft of this post