Keeping Dokku-deployed apps secure

UPDATE 2015-04-03: As it turns out, I was not successfully updating the base Ubuntu image for my app. That aspect of this post has been revised.

I've been doing some playing around with Dokku recently to deploy a private app I've been working on. Despite the fact that it's a bit nitpicky to set up, it's a really great deployment platform. If you're willing to spend a little bit of time setting it up, it's worth it.

However, one thing that's sorely missing from the Dokku docs is the maintenance of the server, specifically how to keep up to date with security issues. With Heartbleed, Shellshock, POODLE, GHOST, and others over the last year, I care a lot about that.

What I've discovered is that there are three levels you need to monitor: your app, the base OS, and your containers.

First, and most obvious, is your app. I'm working on a Rails app, and so a regular gem update; git commit -am 'Update Gemfile' is a necessary maintenance step. What I haven't found yet (if this exists let me know) is something that notifies you when any dependencies in your Gemfile have an available update. If this doesn't exist, you'll get Kudos from me if you build it. If you don't I'll get to it eventually.

Second, there's your host OS. Since Dokku runs on Ubuntu, an aptitude update && aptitude full-upgrade keeps me up to date, and apticron tells me when there's updates to apply. Solved.

For the third, I'm not sure if Dokku provides a tool for this yet (asking on #dokku on Freenode), but you need to update the base image for your container at the same time you update your host OS. This isn't obvious to anyone who hasn't worked with Docker before.

I found there are a few steps to keeping your apps substrate secure:

  1. I needed to install the dokku-rebuild plugin. Not strictly necessary, but it helps.

  2. Whenever apticron notifies me of new packages I need to install on the host, I also run:

    1. docker pull ubuntu:trusty (Dokku is built on Ubuntu Trusty)

    2. dokku rebuild:all (Or git push to Dokku again)

I hope that next time someone out there is Googling for this answer that they find this post and it saves them some time and helps them sleep better at night.

Revival

Well, it's been a really long time. 19 months to be precise. Since my last blog post and since Kyla was born. Coincidence? Not really. Anywho, there's a lot of blog posts in my head to get out, but I thought I'd start with an update:

  • Kyla's 19 months old in less than a week, and she's rambunctious. She's awesome.

  • TribeHR was acquired by NetSuite back in November 2013.

  • We've been growing our team at the NetSuite Waterloo office very rapidly. I now manage a team of 20+.

  • Alex and I bought a house.

  • I don't read Twitter or Facebook anymore, even though I'll post this on Twitter.

  • Alex is now the owner of the Canadian Franchise of Arda Wigs. Awesome quality heat-stylable wigs. Check 'em out.

  • Since being acquired I have traveled to more countries than I've ever been to in my life.

  • I saw Collective Soul, Moist, Slash, Styx, and Aerosmith in concert.

  • In my "spare" time, I think about building great teams and what that entails, as well as software craftsmanship.

  • It's been a wild ride.

I also just got back from the third Hobbit movie. It sucked.

The Gold Standard: Why it's not a measure

You might have thought that with my previous post, this blog would become all about fatherhood or something. Hardly.

I was listening to an Intelligence Squared episode: America doesn't need a strong dollar policy, and found myself incredibly annoyed by one argument that one side used. Worse than that, despite the fact that it's a false analogy, the other side never countered it! This angered me so much, that I have to vent here.

If you want to listen to the debate, now's your last chance before spoilers!

Steve Forbes and James Grant were arguing against the motion, thus in favour of a strong dollar policy. One of their arguments went something like this: If you were a carpenter, and the length of a foot changed from day to day, week to week, month to month, you wouldn't be able to ever know how much wood (in feet) you needed to be able to build a given house. In the same sense, the dollar's value cannot be allowed to change, so that participants in the economy can predictably know how much money they will need to execute a certain transaction. In order to fix the value of money, we peg the exchange rate from dollars to some commodity, say gold (at, for example, $35/ounce).

That sounds great! Now the dollar has a fixed value, and we can all go about our business, right? Wrong. Unlike something like a physical object (to which you can calibrate the measure of a "foot"), the measure of value can change over time. If demand for something increases, some people will be willing to pay more to ensure they get it, and so the price goes up. Same in the opposite direction. A good example is Tickle-Me Elmos. When those hit the market, they cost $28.99 in stores, but people would buy them for (i.e. they were worth) hundreds or thousands of dollars.

The same of course can happen on the supply side. If all the oil wells in the world dried up tomorrow, though demand for oil would be relatively constant, supply would be zero, and again, you'd have a frenzy of people willing to pay anything to get some.

Gold is not immune to the supply/demand pressures either. As a result, its value, relative to other items, can change because of human whim. So fixing the dollar to gold is almost no better than letting it float. Don't be fooled into thinking that the supply and demand for gold are constant, either. When the Spanish brought back vast amounts of gold from the "New World", the supply in Spain skyrocketed, thus the price went down. Instead of being rich, the Spaniards found that because everyone had more gold, things cost more gold to buy!

My point is that the dollar, even just as paper with nothing backing it, is just as good a proxy for value as anything else, even gold. What strikes me as crazy is that Steve Forbes, the man who wrote a book entitled: "Freedom Manifesto: Why Free Markets Are Moral and Big Government Isn’t (2012)", doesn't see the free market involved in the changes in supply and demand for gold, and a reason that a gold standard dollar isn't any better than a floating dollar.

Achievement Unlocked: Procreate!

So... yeah... it's been a year since I've written anything here. Not for the lack of anything to write, more that I'm really bad at keeping up a writing habit. But, with what's happened over the last week, I really can't keep that up anymore.

On Monday, June 3rd, 2013 at 0957h EDT(UTC-0400), Kyla Nerice Gerlach, my daughter, was born to my wife and I. She weighed 7 pounds 14 ounces, was 57 cm tall from head to toe, and was (and still is) our beautifulsea nymph.

Picture time!

Kyla Nerice Gerlach at about 15 minutes old

Having a daughter is... well, it's interesting, mostly. Don't get me wrong, it's amazing, it's hilarious, it's touching, it's all those things, but I think it's interesting most of all. For example, when Kyla first came into the world, she just stared at Alexandra for a long time. She was mesmerized by her Mom's face. She doesn't focus on anything else in the world around her at all, but Mom is captivating. Given that the V1 and V2 visual cortices are not fully developed in newborns, I can't rationalize that it's visual recognition that's happening, but man does it look like it.

Newborns live by their own rules, and like a game of Mao, you have to figure the rules out on your own. At this young age, you're not going to change their rules, all you can do is mold your schedule around them. So far, we're doing pretty good. For example, as I write this I've got Kyla in her car seat (which she loves, go figure) and my wife is in bed. She'll be up again in about 15-20 minutes, which is when I'll head to bed. Gotta finish this post and load the dishwasher before then.

Some people have said that having your first kid is this life-changing moment. I'm not finding that. I'm not a drastically different person than the one I was on Sunday. I know those changes will come over time, but it's been a bit less from an emotional perspective than what I expected. Being someone on anti-depressants, sometimes I wonder if there's something I'm missing as a result of the drugs.

But then I think about how we change as people. I am, in every sense I can think of, a completely different person than I was ten years ago. I wrote crappy code, I was naive about so many things, I didn't have the same handle on my life that I do now. Also, most of the molecules that were in my body then aren't in my body now. Thinking about "self" as a temporally-relevant concept has helped me a lot. Yeah, past-Eric did some stupid stuff, but that was past-Eric. I'm not past-Eric, I'm Eric.

The point I'm getting to is that every day we do wake up as a different person, and though I may not feel the effects of Kyla in my life today, there's been a small inflection in the direction of my life that will alter the makeup of many small changes over time, so future Eric will be a different person than he would have been had Kyla not been born. I think that's what people mean when they say it was a "life-changing moment" in retrospect. One light-year out, a half a degree change in trajectory makes a big difference.

Kyla is stirring. I'm running out of time. Better get to that dishwasher.

Building a SOCKS proxy on EC2 to get around wifi port blocking

At the NXNE Mobile Hackathon, we ran into a small problem. The wifi set up in the room would only allow connections over HTTP and HTTPS, which made it impossible to do many things you might want to do at a hackathon, like:

  1. Push to GitHub over SSH

  2. Connect to MongoDB instances

  3. Connect to... anything... that isn't on ports 80 or 443... so a lot.

If you can configure your tools correctly, the easiest way to get around this kind of problem is via a SOCKS proxy. Normally, I'd set up an SSH tunnel and run the SOCKS proxy over that... but no SSH. So the next best thing is to get a SOCKS server running on EC2. Let's go through the steps required to set this up so that if you end up in the same situation, you can help those around you.

Doing this assumes that you temporarily have an internet connection that is unrestricted, like a tethered smartphone or a wired connection. I'm also assuming that you know your way around EC2 a bit.

  1. Connect to your unrestricted internet connection

  2. Login to EC2

  3. Ensure that you have a keypair setup

  4. Create an EC2 Security Group that opens ports 22 and 443 to the world

  5. Fire up an Ubuntu 12.04 LTS instance (micro will usually do) with your keypair and Security Group

  6. SSH into the new machine with the SSH key (default username: ubuntu)

  7. Run the following commands at the prompt or in a shell script:

    sudo apt-get install build-essential
    wget http://www.inet.no/dante/files/dante-1.3.2.tar.gz # or another version
    tar -zxvf dante-1.3.2.tar.gz
    cd dante-1.3.2
    ./configure
    make
    sudo make install
    
  8. Put the following config in /etc/sock.conf

    ## general configuration (taken from FAQ)
    
    internal: eth0 port = 443
    external: eth0
    method: username none
    user.privileged: root
    user.unprivileged: nobody
    logoutput: stderr
    
    ## client access rules
    
    client pass { from: 0.0.0.0/0 to: 0.0.0.0/0 } # address-range on internal nic.
    
    
    ## server operation access rules
    
    # block connections to localhost, or they will appear to come from the proxy.
    block { from: 0.0.0.0/0 to: lo log: connect }
    
    # allow the rest
    pass { from: 0.0.0.0/0 to: 0.0.0.0/0 }
    
  9. Run sudo sockd -D

Now that we've got the server running, we have to configure our clients to connect to it. Fortunately, this is relatively easy. If you're on linux, run your programs with tsocks. On Windows or Mac, you can try Proxifier (never tried it myself). Remember that the proxy is on port 443.

If you're using PuTTY, you can set your proxy under Connection > Proxy.

This set of steps creates an open proxy that anyone can use to proxy to anywhere. Don't leave it running unless you want really big EC2 bills.

In doing this, I realized that it would be even better to be able to do this via a VPN instead of a SOCKS proxy in order to get better Windows and Mac full capture support. I'm going to play with this idea and post again when I've got something.