Dokku and Firehol

I'm a big fan of both dokku and firehol. Dokku has made my deployments of simple apps near trivial, and firehol is my goto[sic] iptables firewall. My problem is that they don't play well together. I don't think it's dokku's fault, I think the underlying problem is with firehol and docker, but that doesn't really matter to me.

I just upgraded docker, dokku, and iptables on the server I was running, and my firehol configuration stopped working. I don't know why, and I don't care. In the end, this is what worked for me:

What's strange to me is that I need to have the manual DOCKER and DOCKER-ISOLATION chains at the bottom of the configuration, even though to my eyes they don't do anything. My machine does appear to be appropriately secured at the IP packet level though, so I'm happy.

I will talk to some people and see if they know why this is required, though.

Excess Game Codes

Having bought far too many bundles on Steam and Humble Bundle, I have a number of duplicate game codes that I'd like to have go to a good home. If there's a game you want, you will actually play, and are willing to improve my world (an exchange) or the world at large (in some kind of pay-if-forward way), contact me and we'll talk.

Keeping your Dokku-deployed apps secure, take 3

This post I'm writing is mostly in response to the discussion happening on a GitHub issue I filed against dokku. As will all security matters, I've done the best job I can, but it's important to have other eyes look at it too, so if I'm wrong anywhere in here, let me know.

In a previous post, I wrote about keeping the base image under apps deployed on Dokku up to date. When using Dokku, it's important to understand that it's not enough to keep the host OS up to date, you need to keep all the services and libraries installed in all the containers that your app uses up to date as well.

The first area of complexity is that you have two systems to keep up to date: The host OS and the OS inside the container.

The Host OS

Keeping the Host OS up to date is relatively easy in Ubuntu. You can use either unattended-upgrades or cron-apt. Since there's plenty of documentation on the topic, that's all I'll say about that.

The Container OS

Dokku now provides dokku and herokuish packages for dokku, the latter of which theoretically keeps the container environment up to date. Your app containter is build on the herokuish container, which is built on the heroku/cedar:14 container, which is built on the ubuntu-debootstrap container. What this means is that if there's any poor security update policy anywhere in that chain, your app will run insecure software.

The ubuntu-debootstrap container is maintained by a contributor to Docker Hub, and if you look at the history of the manifest, you can see that it isn't updated all that often. The good news is that it doesn't matter! The heroku/cedar:14 image, when it is built, does an apt-get update && apt-get upgrade, so all security updated packages will be brought up to date. This means we're effectively insulated from the poor update cycle on ubuntu-debootstrap.

What we're not isolated from, however, is the poor update cycle on heroku/cedar:14. As of the time of writing, the most recent entry on heroku/cedar:14's build list is from Octover 2nd. Since then, there have been vulnerabilities addressed in OpenJDK, for example. Comparing the GitHub repo history and the heroku/cedar build history it looks like they aren't re-building when there are new security updates, but only when there are new commits.

What this means is that the basis for gliderlabs/herokuish is insecure, and can't be trusted, as the herokuish build process does not do an apt-get update && apt-get upgrade when building.

What can we do?

Manually build the images. :(

Here's the script that I'm now using, updated for the newer version of dokku.

This script is running daily, rebuilding my images to make sure the window of any vulnerabilities on my box are as low as possible.

My vote in the upcoming Canadian election, Part 1 - Bill C-24

I've got a few blog posts brewing, including one about updating Shopify apps, but I need to say a few (read: many) words about the coming Canadian Federal Election. Normally, this would be a series of very nuanced posts about local candidates, their positions, and the positions of the parties that they represent, but the behaviour of parties over the last year has made this election very different from a normal one. Specifically, my vote in this election will be determined by parties voting behaviour on two pieces of legislation. This post is about the first of these.

Bill C-24, the Strengthening Canadian Citizenship Act, received royal assent back in June of 2014. This piece of legislation has a personal impact for me, because my wife and mother-in-law are dual-citizens of Canada and Poland. In a pre-C-24 world, the only way you could have your citizenship revoked was if you had committed fraud when applying. Now there's a litany of "crimes against the state" that can result in the loss of citizenship, including high treason, committing a terrorist act, spying for other countries, or deserting the military.

I don't support high treason, terrorism, or spying for other countries, but I am concerned with things like desertion being in there. Conscientious objection to military service is important to the maintenance of freedom, and exile/deportation/revocation of citizenship is disproportionate punishment for maintaining an ethical stance, especially in a country which has instituted conscription in the past.

Then there's the really strange part. If you're someone who was: "convicted of a terrorism offence as defined in section 2 of the Criminal Code — or an offence outside Canada that, if committed in Canada, would constitute a terrorism offence as defined in that section — and sentenced to at least five years of imprisonment" you can have your citizenship revoked. That's really scary, as it allows a conviction under a system of justice where the standards of evidence and due process are not as high (e.g. North Korea, Myanmar) to allow Canada to revoke your citizenship.

As scary as that is, that's not even the worst part. The worst part is that the process has been "streamlined" to remove the right of the person having their citizenship revoked to have the matter referred to a court. The decision rests solely with the Minister of Citizenship and Immigration. This is the place where I transition from passionate objection to near-rage. To remove a significant amount of judicial process for something as significant as revocation of citizenship is mind boggling (there remains a bit of judicial review, but significantly less than before).

Let me outline a (certainly outlandish) scenario that was not possible before, but could now happen to my wife:

  1. Some future government gets a foreign country who has lax judicial standards and with whom we have an extradition treaty to charge my wife with terrorism;
  2. Allows extradition of my wife to said country, where she is easily convicted;
  3. While in prison there (for, oh, say 5 years), her citizenship is revoked by the Minister of CIC, thus exiling my wife from her family.

Or how about this second scenario:

  1. Some future government wants to fight a war with Poland, and gets conscription instituted;
  2. My wife is conscripted but she refuses to participate;
  3. The Minister of CIC has her citizenship revoked;
  4. She is exiled from Canada, and potentially detained as a POW.

These are far-fetched scenarios, which would never really happen in real life. But what does it say to my wife that Bill C-24 makes these situations possible? It tells her that she is now a different class of citizen from me, the single-citizenship Canadian. Because revoking my citizenship would make me stateless, I'm somehow "more Canadian" than her. It's as if in a post C-24 world, her dual citizenship now makes her "half-Canadian/half-Polish", whereas before she was both 100% Canadian and 100% Polish.

Citizenship is something you don't mess with lightly, and there are holes in this attempt to "protect" the concept of Canadian Citizenship. There are plenty of undesirable people who are Canadian citizens. They do horrible things. In a democratic society, we don't deal with them by kicking them out. I think the message about our country would be more positive if we were willing to clean up our own messes, rather than claiming they're not our problem anymore.

There's plenty else wrong with this bill, and a number of good things in there too. The problem I have described above is what makes it completely unethical to me and in my mind anathema to the concept of citizenship.

Because the Conservative party drafted and voted for this bill (with the whip), I cannot ethically support them in the coming election.

In the next post, I'll explore the second piece of legislation that makes this election easy for me, and then I'll talk about what this means for my vote.

P.S. If you want to compare the relevant section of the law, you can look at the old vs. the new.

Keeping your Dokku-deployed apps secure, revised

Updated 2015-08-30: Dokku has changed its stack to include herokuish instead of buildstep. This makes things better as will be coming very soon in a new blog post.

As it turns out, I had keeping my dokku apps up to date completely wrong. The containers that your dokku apps run in are not based on the ubuntu:trusty Docker image. They're instead based on ubuntu-debootstrap:14.04. Additionally, currently you can't trust the progrium/cedarish and progrium/buildstep images from Docker Hub, as they're not updated when the base image is updated (Issues are filed on cedarish and buildstep to make this rebuilding automatic on Docker Hub).

However, you can tell your host machine to rebuild the images itself. The script I'm now running daily to keep all the dokku things up to date is below:

  • Lines 8 and 9 pull the latest Ubuntu images from Docker Hub.
  • Lines 11-23 update all the dokku plugins I have. This is an optional step, especially to be avoided if you need to vet every change to your environment.
  • Lines 26-29 rebuild the cedarish image. Note that this will only do a build if the base image is also new. Docker is good about this.
  • Lines 32-25 do the same thing for the buildstep image.
  • Line 38 rebuilds and redeploys all of our apps.
  • Line 39 waits for 2 minutes, so that the old containers can die peacefully.
  • Lines 42 and 43 clean up old containers and images. There's some good discussion on Docker container cleanup methods, I picked what I liked.


  • Detect if either cedarish or buildstep actually changed in their rebuilds, and exit before line 38 if they did.