Keeping your Dokku-deployed apps secure, take 3

This post I'm writing is mostly in response to the discussion happening on a GitHub issue I filed against dokku. As will all security matters, I've done the best job I can, but it's important to have other eyes look at it too, so if I'm wrong anywhere in here, let me know.

In a previous post, I wrote about keeping the base image under apps deployed on Dokku up to date. When using Dokku, it's important to understand that it's not enough to keep the host OS up to date, you need to keep all the services and libraries installed in all the containers that your app uses up to date as well.

The first area of complexity is that you have two systems to keep up to date: The host OS and the OS inside the container.

The Host OS

Keeping the Host OS up to date is relatively easy in Ubuntu. You can use either unattended-upgrades or cron-apt. Since there's plenty of documentation on the topic, that's all I'll say about that.

The Container OS

Dokku now provides dokku and herokuish packages for dokku, the latter of which theoretically keeps the container environment up to date. Your app containter is build on the herokuish container, which is built on the heroku/cedar:14 container, which is built on the ubuntu-debootstrap container. What this means is that if there's any poor security update policy anywhere in that chain, your app will run insecure software.

The ubuntu-debootstrap container is maintained by a contributor to Docker Hub, and if you look at the history of the manifest, you can see that it isn't updated all that often. The good news is that it doesn't matter! The heroku/cedar:14 image, when it is built, does an apt-get update && apt-get upgrade, so all security updated packages will be brought up to date. This means we're effectively insulated from the poor update cycle on ubuntu-debootstrap.

What we're not isolated from, however, is the poor update cycle on heroku/cedar:14. As of the time of writing, the most recent entry on heroku/cedar:14's build list is from Octover 2nd. Since then, there have been vulnerabilities addressed in OpenJDK, for example. Comparing the GitHub repo history and the heroku/cedar build history it looks like they aren't re-building when there are new security updates, but only when there are new commits.

What this means is that the basis for gliderlabs/herokuish is insecure, and can't be trusted, as the herokuish build process does not do an apt-get update && apt-get upgrade when building.

What can we do?

Manually build the images. :(

Here's the script that I'm now using, updated for the newer version of dokku.

This script is running daily, rebuilding my images to make sure the window of any vulnerabilities on my box are as low as possible.

My vote in the upcoming Canadian election, Part 1 - Bill C-24

I've got a few blog posts brewing, including one about updating Shopify apps, but I need to say a few (read: many) words about the coming Canadian Federal Election. Normally, this would be a series of very nuanced posts about local candidates, their positions, and the positions of the parties that they represent, but the behaviour of parties over the last year has made this election very different from a normal one. Specifically, my vote in this election will be determined by parties voting behaviour on two pieces of legislation. This post is about the first of these.

Bill C-24, the Strengthening Canadian Citizenship Act, received royal assent back in June of 2014. This piece of legislation has a personal impact for me, because my wife and mother-in-law are dual-citizens of Canada and Poland. In a pre-C-24 world, the only way you could have your citizenship revoked was if you had committed fraud when applying. Now there's a litany of "crimes against the state" that can result in the loss of citizenship, including high treason, committing a terrorist act, spying for other countries, or deserting the military.

I don't support high treason, terrorism, or spying for other countries, but I am concerned with things like desertion being in there. Conscientious objection to military service is important to the maintenance of freedom, and exile/deportation/revocation of citizenship is disproportionate punishment for maintaining an ethical stance, especially in a country which has instituted conscription in the past.

Then there's the really strange part. If you're someone who was: "convicted of a terrorism offence as defined in section 2 of the Criminal Code — or an offence outside Canada that, if committed in Canada, would constitute a terrorism offence as defined in that section — and sentenced to at least five years of imprisonment" you can have your citizenship revoked. That's really scary, as it allows a conviction under a system of justice where the standards of evidence and due process are not as high (e.g. North Korea, Myanmar) to allow Canada to revoke your citizenship.

As scary as that is, that's not even the worst part. The worst part is that the process has been "streamlined" to remove the right of the person having their citizenship revoked to have the matter referred to a court. The decision rests solely with the Minister of Citizenship and Immigration. This is the place where I transition from passionate objection to near-rage. To remove a significant amount of judicial process for something as significant as revocation of citizenship is mind boggling (there remains a bit of judicial review, but significantly less than before).

Let me outline a (certainly outlandish) scenario that was not possible before, but could now happen to my wife:

  1. Some future government gets a foreign country who has lax judicial standards and with whom we have an extradition treaty to charge my wife with terrorism;
  2. Allows extradition of my wife to said country, where she is easily convicted;
  3. While in prison there (for, oh, say 5 years), her citizenship is revoked by the Minister of CIC, thus exiling my wife from her family.

Or how about this second scenario:

  1. Some future government wants to fight a war with Poland, and gets conscription instituted;
  2. My wife is conscripted but she refuses to participate;
  3. The Minister of CIC has her citizenship revoked;
  4. She is exiled from Canada, and potentially detained as a POW.

These are far-fetched scenarios, which would never really happen in real life. But what does it say to my wife that Bill C-24 makes these situations possible? It tells her that she is now a different class of citizen from me, the single-citizenship Canadian. Because revoking my citizenship would make me stateless, I'm somehow "more Canadian" than her. It's as if in a post C-24 world, her dual citizenship now makes her "half-Canadian/half-Polish", whereas before she was both 100% Canadian and 100% Polish.

Citizenship is something you don't mess with lightly, and there are holes in this attempt to "protect" the concept of Canadian Citizenship. There are plenty of undesirable people who are Canadian citizens. They do horrible things. In a democratic society, we don't deal with them by kicking them out. I think the message about our country would be more positive if we were willing to clean up our own messes, rather than claiming they're not our problem anymore.

There's plenty else wrong with this bill, and a number of good things in there too. The problem I have described above is what makes it completely unethical to me and in my mind anathema to the concept of citizenship.

Because the Conservative party drafted and voted for this bill (with the whip), I cannot ethically support them in the coming election.

In the next post, I'll explore the second piece of legislation that makes this election easy for me, and then I'll talk about what this means for my vote.

P.S. If you want to compare the relevant section of the law, you can look at the old vs. the new.

Keeping your Dokku-deployed apps secure, revised

Updated 2015-08-30: Dokku has changed its stack to include herokuish instead of buildstep. This makes things better as will be coming very soon in a new blog post.

As it turns out, I had keeping my dokku apps up to date completely wrong. The containers that your dokku apps run in are not based on the ubuntu:trusty Docker image. They're instead based on ubuntu-debootstrap:14.04. Additionally, currently you can't trust the progrium/cedarish and progrium/buildstep images from Docker Hub, as they're not updated when the base image is updated (Issues are filed on cedarish and buildstep to make this rebuilding automatic on Docker Hub).

However, you can tell your host machine to rebuild the images itself. The script I'm now running daily to keep all the dokku things up to date is below:

  • Lines 8 and 9 pull the latest Ubuntu images from Docker Hub.
  • Lines 11-23 update all the dokku plugins I have. This is an optional step, especially to be avoided if you need to vet every change to your environment.
  • Lines 26-29 rebuild the cedarish image. Note that this will only do a build if the base image is also new. Docker is good about this.
  • Lines 32-25 do the same thing for the buildstep image.
  • Line 38 rebuilds and redeploys all of our apps.
  • Line 39 waits for 2 minutes, so that the old containers can die peacefully.
  • Lines 42 and 43 clean up old containers and images. There's some good discussion on Docker container cleanup methods, I picked what I liked.


  • Detect if either cedarish or buildstep actually changed in their rebuilds, and exit before line 38 if they did.

Keeping Dokku-deployed apps secure

UPDATE 2015-04-03: As it turns out, I was not successfully updating the base Ubuntu image for my app. That aspect of this post has been revised.

I've been doing some playing around with Dokku recently to deploy a private app I've been working on. Despite the fact that it's a bit nitpicky to set up, it's a really great deployment platform. If you're willing to spend a little bit of time setting it up, it's worth it.

However, one thing that's sorely missing from the Dokku docs is the maintenance of the server, specifically how to keep up to date with security issues. With Heartbleed, Shellshock, POODLE, GHOST, and others over the last year, I care a lot about that.

What I've discovered is that there are three levels you need to monitor: your app, the base OS, and your containers.

First, and most obvious, is your app. I'm working on a Rails app, and so a regular gem update; git commit -am 'Update Gemfile' is a necessary maintenance step. What I haven't found yet (if this exists let me know) is something that notifies you when any dependencies in your Gemfile have an available update. If this doesn't exist, you'll get Kudos from me if you build it. If you don't I'll get to it eventually.

Second, there's your host OS. Since Dokku runs on Ubuntu, an aptitude update && aptitude full-upgrade keeps me up to date, and apticron tells me when there's updates to apply. Solved.

For the third, I'm not sure if Dokku provides a tool for this yet (asking on #dokku on Freenode), but you need to update the base image for your container at the same time you update your host OS. This isn't obvious to anyone who hasn't worked with Docker before.

I found there are a few steps to keeping your apps substrate secure:

  1. I needed to install the dokku-rebuild plugin. Not strictly necessary, but it helps.
  2. Whenever apticron notifies me of new packages I need to install on the host, I also run:
    1. docker pull ubuntu:trusty (Dokku is built on Ubuntu Trusty)
    2. dokku rebuild:all (Or git push to Dokku again)

I hope that next time someone out there is Googling for this answer that they find this post and it saves them some time and helps them sleep better at night.


Well, it's been a really long time. 19 months to be precise. Since my last blog post and since Kyla was born. Coincidence? Not really. Anywho, there's a lot of blog posts in my head to get out, but I thought I'd start with an update:

  • Kyla's 19 months old in less than a week, and she's rambunctious. She's awesome.
  • TribeHR was acquired by NetSuite back in November 2013.
  • We've been growing our team at the NetSuite Waterloo office very rapidly. I now manage a team of 20+.
  • Alex and I bought a house.
  • I don't read Twitter or Facebook anymore, even though I'll post this on Twitter.
  • Alex is now the owner of the Canadian Franchise of Arda Wigs. Awesome quality heat-stylable wigs. Check 'em out.
  • Since being acquired I have traveled to more countries than I've ever been to in my life.
  • I saw Collective Soul, Moist, Slash, Styx, and Aerosmith in concert.
  • In my "spare" time, I think about building great teams and what that entails, as well as software craftsmanship.
  • It's been a wild ride.

I also just got back from the third Hobbit movie. It sucked.