Dokku and Firehol

I'm a big fan of both dokku and firehol. Dokku has made my deployments of simple apps near trivial, and firehol is my goto[sic] iptables firewall. My problem is that they don't play well together. I don't think it's dokku's fault, I think the underlying problem is with firehol and docker, but that doesn't really matter to me.

I just upgraded docker, dokku, and iptables on the server I was running, and my firehol configuration stopped working. I don't know why, and I don't care. In the end, this is what worked for me:

What's strange to me is that I need to have the manual DOCKER and DOCKER-ISOLATION chains at the bottom of the configuration, even though to my eyes they don't do anything. My machine does appear to be appropriately secured at the IP packet level though, so I'm happy.

I will talk to some people and see if they know why this is required, though.

Keeping your Dokku-deployed apps secure, revised

Updated 2015-08-30: Dokku has changed its stack to include herokuish instead of buildstep. This makes things better as will be coming very soon in a new blog post.

As it turns out, I had keeping my dokku apps up to date completely wrong. The containers that your dokku apps run in are not based on the ubuntu:trusty Docker image. They're instead based on ubuntu-debootstrap:14.04. Additionally, currently you can't trust the progrium/cedarish and progrium/buildstep images from Docker Hub, as they're not updated when the base image is updated (Issues are filed on cedarish and buildstep to make this rebuilding automatic on Docker Hub).

However, you can tell your host machine to rebuild the images itself. The script I'm now running daily to keep all the dokku things up to date is below:

  • Lines 8 and 9 pull the latest Ubuntu images from Docker Hub.
  • Lines 11-23 update all the dokku plugins I have. This is an optional step, especially to be avoided if you need to vet every change to your environment.
  • Lines 26-29 rebuild the cedarish image. Note that this will only do a build if the base image is also new. Docker is good about this.
  • Lines 32-25 do the same thing for the buildstep image.
  • Line 38 rebuilds and redeploys all of our apps.
  • Line 39 waits for 2 minutes, so that the old containers can die peacefully.
  • Lines 42 and 43 clean up old containers and images. There's some good discussion on Docker container cleanup methods, I picked what I liked.

TODO:

  • Detect if either cedarish or buildstep actually changed in their rebuilds, and exit before line 38 if they did.

Keeping Dokku-deployed apps secure

UPDATE 2015-04-03: As it turns out, I was not successfully updating the base Ubuntu image for my app. That aspect of this post has been revised.

I've been doing some playing around with Dokku recently to deploy a private app I've been working on. Despite the fact that it's a bit nitpicky to set up, it's a really great deployment platform. If you're willing to spend a little bit of time setting it up, it's worth it.

However, one thing that's sorely missing from the Dokku docs is the maintenance of the server, specifically how to keep up to date with security issues. With Heartbleed, Shellshock, POODLE, GHOST, and others over the last year, I care a lot about that.

What I've discovered is that there are three levels you need to monitor: your app, the base OS, and your containers.

First, and most obvious, is your app. I'm working on a Rails app, and so a regular gem update; git commit -am 'Update Gemfile' is a necessary maintenance step. What I haven't found yet (if this exists let me know) is something that notifies you when any dependencies in your Gemfile have an available update. If this doesn't exist, you'll get Kudos from me if you build it. If you don't I'll get to it eventually.

Second, there's your host OS. Since Dokku runs on Ubuntu, an aptitude update && aptitude full-upgrade keeps me up to date, and apticron tells me when there's updates to apply. Solved.

For the third, I'm not sure if Dokku provides a tool for this yet (asking on #dokku on Freenode), but you need to update the base image for your container at the same time you update your host OS. This isn't obvious to anyone who hasn't worked with Docker before.

I found there are a few steps to keeping your apps substrate secure:

  1. I needed to install the dokku-rebuild plugin. Not strictly necessary, but it helps.

  2. Whenever apticron notifies me of new packages I need to install on the host, I also run:

    1. docker pull ubuntu:trusty (Dokku is built on Ubuntu Trusty)

    2. dokku rebuild:all (Or git push to Dokku again)

I hope that next time someone out there is Googling for this answer that they find this post and it saves them some time and helps them sleep better at night.