Keeping your Dokku-deployed apps secure, revised

Updated 2015-08-30: Dokku has changed its stack to include herokuish instead of buildstep. This makes things better as will be coming very soon in a new blog post.

As it turns out, I had keeping my dokku apps up to date completely wrong. The containers that your dokku apps run in are not based on the ubuntu:trusty Docker image. They're instead based on ubuntu-debootstrap:14.04. Additionally, currently you can't trust the progrium/cedarish and progrium/buildstep images from Docker Hub, as they're not updated when the base image is updated (Issues are filed on cedarish and buildstep to make this rebuilding automatic on Docker Hub).

However, you can tell your host machine to rebuild the images itself. The script I'm now running daily to keep all the dokku things up to date is below:

• Lines 8 and 9 pull the latest Ubuntu images from Docker Hub.
• Lines 11-23 update all the dokku plugins I have. This is an optional step, especially to be avoided if you need to vet every change to your environment.
• Lines 26-29 rebuild the cedarish image. Note that this will only do a build if the base image is also new. Docker is good about this.
• Lines 32-25 do the same thing for the buildstep image.
• Line 38 rebuilds and redeploys all of our apps.
• Line 39 waits for 2 minutes, so that the old containers can die peacefully.
• Lines 42 and 43 clean up old containers and images. There's some good discussion on Docker container cleanup methods, I picked what I liked.

TODO:

• Detect if either cedarish or buildstep actually changed in their rebuilds, and exit before line 38 if they did.

Keeping Dokku-deployed apps secure

UPDATE 2015-04-03: As it turns out, I was not successfully updating the base Ubuntu image for my app. That aspect of this post has been revised.

I've been doing some playing around with Dokku recently to deploy a private app I've been working on. Despite the fact that it's a bit nitpicky to set up, it's a really great deployment platform. If you're willing to spend a little bit of time setting it up, it's worth it.

However, one thing that's sorely missing from the Dokku docs is the maintenance of the server, specifically how to keep up to date with security issues. With Heartbleed, Shellshock, POODLE, GHOST, and others over the last year, I care a lot about that.

What I've discovered is that there are three levels you need to monitor: your app, the base OS, and your containers.

First, and most obvious, is your app. I'm working on a Rails app, and so a regular gem update; git commit -am 'Update Gemfile' is a necessary maintenance step. What I haven't found yet (if this exists let me know) is something that notifies you when any dependencies in your Gemfile have an available update. If this doesn't exist, you'll get Kudos from me if you build it. If you don't I'll get to it eventually.

Second, there's your host OS. Since Dokku runs on Ubuntu, an aptitude update && aptitude full-upgrade keeps me up to date, and apticron tells me when there's updates to apply. Solved.

For the third, I'm not sure if Dokku provides a tool for this yet (asking on #dokku on Freenode), but you need to update the base image for your container at the same time you update your host OS. This isn't obvious to anyone who hasn't worked with Docker before.

I found there are a few steps to keeping your apps substrate secure:

1. I needed to install the dokku-rebuild plugin. Not strictly necessary, but it helps.

2. Whenever apticron notifies me of new packages I need to install on the host, I also run:

   1. docker pull ubuntu:trusty (Dokku is built on Ubuntu Trusty)

   2. dokku rebuild:all (Or git push to Dokku again)

I hope that next time someone out there is Googling for this answer that they find this post and it saves them some time and helps them sleep better at night.

Revival

Well, it's been a really long time. 19 months to be precise. Since my last blog post and since Kyla was born. Coincidence? Not really. Anywho, there's a lot of blog posts in my head to get out, but I thought I'd start with an update:

• Kyla's 19 months old in less than a week, and she's rambunctious. She's awesome.

• TribeHR was acquired by NetSuite back in November 2013.

• We've been growing our team at the NetSuite Waterloo office very rapidly. I now manage a team of 20+.

• Alex and I bought a house.

• I don't read Twitter or Facebook anymore, even though I'll post this on Twitter.

• Alex is now the owner of the Canadian Franchise of Arda Wigs. Awesome quality heat-stylable wigs. Check 'em out.

• Since being acquired I have traveled to more countries than I've ever been to in my life.

• I saw Collective Soul, Moist, Slash, Styx, and Aerosmith in concert.

• In my "spare" time, I think about building great teams and what that entails, as well as software craftsmanship.

• It's been a wild ride.

I also just got back from the third Hobbit movie. It sucked.

The Gold Standard: Why it's not a measure

You might have thought that with my previous post, this blog would become all about fatherhood or something. Hardly.

I was listening to an Intelligence Squared episode: America doesn't need a strong dollar policy, and found myself incredibly annoyed by one argument that one side used. Worse than that, despite the fact that it's a false analogy, the other side never countered it! This angered me so much, that I have to vent here.

If you want to listen to the debate, now's your last chance before spoilers!

Steve Forbes and James Grant were arguing against the motion, thus in favour of a strong dollar policy. One of their arguments went something like this: If you were a carpenter, and the length of a foot changed from day to day, week to week, month to month, you wouldn't be able to ever know how much wood (in feet) you needed to be able to build a given house. In the same sense, the dollar's value cannot be allowed to change, so that participants in the economy can predictably know how much money they will need to execute a certain transaction. In order to fix the value of money, we peg the exchange rate from dollars to some commodity, say gold (at, for example, $35/ounce).

That sounds great! Now the dollar has a fixed value, and we can all go about our business, right? Wrong. Unlike something like a physical object (to which you can calibrate the measure of a "foot"), the measure of value can change over time. If demand for something increases, some people will be willing to pay more to ensure they get it, and so the price goes up. Same in the opposite direction. A good example is Tickle-Me Elmos. When those hit the market, they cost $28.99 in stores, but people would buy them for (i.e. they were worth) hundreds or thousands of dollars.

The same of course can happen on the supply side. If all the oil wells in the world dried up tomorrow, though demand for oil would be relatively constant, supply would be zero, and again, you'd have a frenzy of people willing to pay anything to get some.

Gold is not immune to the supply/demand pressures either. As a result, its value, relative to other items, can change because of human whim. So fixing the dollar to gold is almost no better than letting it float. Don't be fooled into thinking that the supply and demand for gold are constant, either. When the Spanish brought back vast amounts of gold from the "New World", the supply in Spain skyrocketed, thus the price went down. Instead of being rich, the Spaniards found that because everyone had more gold, things cost more gold to buy!

My point is that the dollar, even just as paper with nothing backing it, is just as good a proxy for value as anything else, even gold. What strikes me as crazy is that Steve Forbes, the man who wrote a book entitled: "Freedom Manifesto: Why Free Markets Are Moral and Big Government Isn’t (2012)", doesn't see the free market involved in the changes in supply and demand for gold, and a reason that a gold standard dollar isn't any better than a floating dollar.

Achievement Unlocked: Procreate!

So... yeah... it's been a year since I've written anything here. Not for the lack of anything to write, more that I'm really bad at keeping up a writing habit. But, with what's happened over the last week, I really can't keep that up anymore.

On Monday, June 3rd, 2013 at 0957h EDT(UTC-0400), Kyla Nerice Gerlach, my daughter, was born to my wife and I. She weighed 7 pounds 14 ounces, was 57 cm tall from head to toe, and was (and still is) our beautifulsea nymph.

Picture time!

Having a daughter is... well, it's interesting, mostly. Don't get me wrong, it's amazing, it's hilarious, it's touching, it's all those things, but I think it's interesting most of all. For example, when Kyla first came into the world, she just stared at Alexandra for a long time. She was mesmerized by her Mom's face. She doesn't focus on anything else in the world around her at all, but Mom is captivating. Given that the V1 and V2 visual cortices are not fully developed in newborns, I can't rationalize that it's visual recognition that's happening, but man does it look like it.

Newborns live by their own rules, and like a game of Mao, you have to figure the rules out on your own. At this young age, you're not going to change their rules, all you can do is mold your schedule around them. So far, we're doing pretty good. For example, as I write this I've got Kyla in her car seat (which she loves, go figure) and my wife is in bed. She'll be up again in about 15-20 minutes, which is when I'll head to bed. Gotta finish this post and load the dishwasher before then.

Some people have said that having your first kid is this life-changing moment. I'm not finding that. I'm not a drastically different person than the one I was on Sunday. I know those changes will come over time, but it's been a bit less from an emotional perspective than what I expected. Being someone on anti-depressants, sometimes I wonder if there's something I'm missing as a result of the drugs.

But then I think about how we change as people. I am, in every sense I can think of, a completely different person than I was ten years ago. I wrote crappy code, I was naive about so many things, I didn't have the same handle on my life that I do now. Also, most of the molecules that were in my body then aren't in my body now. Thinking about "self" as a temporally-relevant concept has helped me a lot. Yeah, past-Eric did some stupid stuff, but that was past-Eric. I'm not past-Eric, I'm Eric.

The point I'm getting to is that every day we do wake up as a different person, and though I may not feel the effects of Kyla in my life today, there's been a small inflection in the direction of my life that will alter the makeup of many small changes over time, so future Eric will be a different person than he would have been had Kyla not been born. I think that's what people mean when they say it was a "life-changing moment" in retrospect. One light-year out, a half a degree change in trajectory makes a big difference.

Kyla is stirring. I'm running out of time. Better get to that dishwasher.