Keeping your Dokku-deployed apps secure, revised

Updated 2015-08-30: Dokku has changed its stack to include herokuish instead of buildstep. This makes things better as will be coming very soon in a new blog post.

As it turns out, I had keeping my dokku apps up to date completely wrong. The containers that your dokku apps run in are not based on the ubuntu:trusty Docker image. They're instead based on ubuntu-debootstrap:14.04. Additionally, currently you can't trust the progrium/cedarish and progrium/buildstep images from Docker Hub, as they're not updated when the base image is updated (Issues are filed on cedarish and buildstep to make this rebuilding automatic on Docker Hub).

However, you can tell your host machine to rebuild the images itself. The script I'm now running daily to keep all the dokku things up to date is below:

• Lines 8 and 9 pull the latest Ubuntu images from Docker Hub.
• Lines 11-23 update all the dokku plugins I have. This is an optional step, especially to be avoided if you need to vet every change to your environment.
• Lines 26-29 rebuild the cedarish image. Note that this will only do a build if the base image is also new. Docker is good about this.
• Lines 32-25 do the same thing for the buildstep image.
• Line 38 rebuilds and redeploys all of our apps.
• Line 39 waits for 2 minutes, so that the old containers can die peacefully.
• Lines 42 and 43 clean up old containers and images. There's some good discussion on Docker container cleanup methods, I picked what I liked.

TODO:

• Detect if either cedarish or buildstep actually changed in their rebuilds, and exit before line 38 if they did.

Keeping Dokku-deployed apps secure

UPDATE 2015-04-03: As it turns out, I was not successfully updating the base Ubuntu image for my app. That aspect of this post has been revised.

I've been doing some playing around with Dokku recently to deploy a private app I've been working on. Despite the fact that it's a bit nitpicky to set up, it's a really great deployment platform. If you're willing to spend a little bit of time setting it up, it's worth it.

However, one thing that's sorely missing from the Dokku docs is the maintenance of the server, specifically how to keep up to date with security issues. With Heartbleed, Shellshock, POODLE, GHOST, and others over the last year, I care a lot about that.

What I've discovered is that there are three levels you need to monitor: your app, the base OS, and your containers.

First, and most obvious, is your app. I'm working on a Rails app, and so a regular gem update; git commit -am 'Update Gemfile' is a necessary maintenance step. What I haven't found yet (if this exists let me know) is something that notifies you when any dependencies in your Gemfile have an available update. If this doesn't exist, you'll get Kudos from me if you build it. If you don't I'll get to it eventually.

Second, there's your host OS. Since Dokku runs on Ubuntu, an aptitude update && aptitude full-upgrade keeps me up to date, and apticron tells me when there's updates to apply. Solved.

For the third, I'm not sure if Dokku provides a tool for this yet (asking on #dokku on Freenode), but you need to update the base image for your container at the same time you update your host OS. This isn't obvious to anyone who hasn't worked with Docker before.

I found there are a few steps to keeping your apps substrate secure:

1. I needed to install the dokku-rebuild plugin. Not strictly necessary, but it helps.

2. Whenever apticron notifies me of new packages I need to install on the host, I also run:

   1. docker pull ubuntu:trusty (Dokku is built on Ubuntu Trusty)

   2. dokku rebuild:all (Or git push to Dokku again)

I hope that next time someone out there is Googling for this answer that they find this post and it saves them some time and helps them sleep better at night.

Some of the most wonderful things I've ever seen

First of all, credit where credit is due. This comes by way of @thinkontheclock via @josephfung (over a week ago). I spooled it, and just watched it tonight. If you've got 35 minutes, watch the first half of this talk. If you have less, I'll tell you where to start).

If you're a software developer of any kind, start at 10:20. If you see the GUI, and you're like "Yeah, that's nice, but how do I work with real code?" go to 16:25.

If you're an electrical engineer, start at 23:00.

If you're an artist, start at 29:20.

Don't wait if you can avoid it. Watch it now.

Bret Victor - Inventing on Principle from CUSEC on Vimeo.

If what you saw in those few minutes (of whatever segment you watched) impressed you, watch the first 35 minutes in its entirety. It's amazing, even if you don't exactly know what's going on every second.

If you want to see some of this in action, check out Bret's site, specifically the page on Tangle. (I'll forgive the hashbang URL this once, because you're awesome, Bret.)

Breaking the blog

So, I had noticed that some of the modules I had installed on this Drupal-based blog were out of date. Thinking it might be a problem with my Drupaldeb service, I started there. First, it appears that I might have a few bugs with ensuring that packages are actually properly put into the APT repository. There were two packages that were out of sync between the website and the repo. Need to fix that in the future.

But then I found the real problem. When you upgrade Ubuntu, it disables other APT sources. So when I last updated, it disabled the Drupaldeb APT repo, thus not allowing me updates. Simple task to re-enable that and voila. Modules updating.

After I had everything updated, I needed to update the database. That's when I made the fatal flaw of running drush up instead of drush updatedb. The former command happily overwrite my APT installed version of Drupal with 6.25 from the repository, which disabled a bunch of my modules. They even disappeared from my list of installed but disabled modules! Oh no, my website looks like crap!

After manually re-enabling the modules in the database (which didn't work), and downgrading Drupal again (which didn't work), I found the problem. When Drupal upgraded, it deleted everything from my /usr/share/drupal6 directory. Including my modules. So all I had to do was reinstall all my modules and themes and I was fine. This is something else I will have to test soon.

Getting videos offline, automatically, on my phone

There's several video shows that I like to follow: Zero Punctuation and Extra Credits, for example. I like to get the newest episode automatically. Up until a few months ago, I was content to have them pulled into Google Reader, and then to watch them as time permitted. It worked okay, but they would tend to pile up. Sometimes I'd try to watch them on my mobile, but the buffering meant I wouldn't get too many in on a bus ride.

Then I discovered Spool. Spool allows you to record videos and have them downloaded to your phone for later watching. Oh happy day! My pattern then became:

1. Every day, go to Google Reader.

2. Click "Add to Spool" for each video entry.

3. Watch at my leisure.

This past weekend, I realized something. RSS is designed to be automated. (Duh). So I wrote myself a plugin for FlexGet that takes the URL from an RSS feed and sends it off to Spool. Now the videos just show up in My Spool as they are posted!

So yeah, it's available on GitHub: https://github.com/egerlach/flexget-plugins Go ahead and download, fork, and submit Pull Requests. It seems to only work right now if you copy it into the FlexGet main plugins directory... I can't seem to get it to work at ~/.flexget/plugins. That's the next step. (UPDATE: It now works in ~/.flexget/plugins! I just couldn't have it in ~/.flexget/plugins/output is all)

The only problem now is that on Tuesday, YouTube asked Spool to not record their videos and make them available offline. This is annoying, because now the shows I watch from YouTube I can't get offline. I'll just have to write another plugin for that...