Posts Archive

In the first post on how we moved our application to Docker on AWS, I described the reason for us to do it, and how we set up the servers. This post describes how we’ve organized the different Docker containers.

Container Setup On AWS

Our setup on AWS looks like this:

We use an EC2 load balancer to route traffic to our web hosts. This is an easy way to setup clustering, but you should be aware that EC2 does not allow for any alternative route or html page if no hosts are available. This means that if all instances are down for some reason, the only thing shown is a blank page. This has been a long standing feature request which for some reason hasn’t been implemented yet.

Last fall, we moved the Qwaya Facebook ads tool from a single host on Rackspace to a clustered Docker based environment on AWS. This is the first blog post of hopefully more that I’m writing to remember what we did. I’ve planned two more posts, one on how we handle CI and one on container orchestration.

Why Docker?

The task at hand was not “Move to docker”, it was “get out of single host hosting”. The old deployment consisted of some quite complicated Fabric scripts that was run from an arbitrary developers laptop. The source code repository did not match the deployment setup, so there was a lot of remote copy commands running around. I know that this is a very common way of deploying applications, but it brings a number of problems. For example, there are no build artifacts, making repeatable builds harder, and if the build fails halfway in you will have a broken system.

At Qwaya we run our servers in AWS. We build our own base AMI’s in order to ensure a stable base infrastructure and to be able to quickly launch new servers when needed.

We have a few AMI’s arranged as described in the image below:

• The base image holds all the packages we want installed on all hosts, it sets up docker, logspout logging to Papertrail and [Datadog][datadog] monitoring.
• The Web and Worker images holds different systemd files for different parts of our application.
• We use Buildkite for building and deploying, which uses a bring-your-own technique for build servers. We have two kinds, one for building and one for deploying.

While we really should have our configs in something like Consul, we’re not there at the moment. In order to limit the number of AMI rebuilds, we try to place configs in cloud-init to limit rebuilds.

Setting up the integriy.st OpenVPN on NetworkManager is easier than the official docs describe. Instead of manually entering all the data, you can use the client.ovpn file included in the .zip downloaded.

When creating the OpenVPN connection, select “Import a saved OpenVPN configuration …”.

Then select the client.ovpn file. You probably want a better connection name than “client” so make one up and fill it in. Enter your username and password, save and connect.

My daily yaourt -Syua led me to realize that fig has been deprecated in favor of docker-compose. Switching however was just a straight renaming exercise. The fig.yml file is now docker-compose.yml, and the command is docker-compose instead of fig

I just released 0.3.0 of my static blogging tool Sinor.

Major new features in this release:

• Tags support
• Markdown can be used in archive pages
• Improved command line API

It can be installed either by pip install sinor or by downloading it directly from Github.

I started blogging back in 2002. At that time I was working at Lecando, where I was lucky enough to be doing Java development and experimenting with XP together with some ridiculously bright people like Jimmy, Johan, Jon, Håkan and Visar.

Back then, Java was where it was happening. All the new things happened in Javaland, and we did our best to try out the new things; TDD with JUnit, CI with CruiseControl.

I spent a lot of time then reading blogs. Most of the blogs I was interested in were hosted at FreeRoller, a platform run by Anthony Eden, today perhaps more famous for DNSimple.

Almost a year and a half ago, I wrote about how I had built my own blog frameworky thing. In that post I wrote that I was happy with the result. Unfortunately, it didn’t last long.

The problem was that the tool I had built did too much. It found all the markdown files, converted them to HTML using Mustache templates and wrote the result to other files. All paths etcetera were hardcoded, the code was tied to the content in a way I strongly diskliked. Each change required everything to be rebuilt. Come to think of it, the only thing I liked was the choices of templating technology I had picked - Markdown and Mustache.

Friday was my last day as a consultant at Valtech, and starting Monday I will join the team of developers at Qwaya.

This was the second time I was employed by Valtech. I originally joined them in 2005 after five years at Lecando. Back then, Valtech was one of only three companies in Stockholm actively promoting agile software development. (The other two was Agero and Citerus). I spent two years away from Valtech, first in product development at WeMind and after that trying to set up a Thoughtworks office in Stockholm. Neither of those two turned out the way I had hoped, so I was happy to return to Valtech in the fall of 2009.

Recently I was replying to a post on the Valtech intranet, and tried coming up with a word for what I do as an agile coach. The thing that popped up in my mind was agile smurfing, which I think nails it. As an agile coach I do everything between teaching TDD to solving organizational issues, and since you can exchange every word in the smurf language to “smurf” it just works.