Andrew Perkins
tech i like

Getting comfortable with Kamal

I have tried out a whole bunch of infrastructure tools over the years, with mixed results. Typically I am deploying web servers, with a varying degree of complexity.

Plenty of the servers that I deploy are very simple one-off servers for a particular exhibition or show. I also maintain a lot of systems, most of which I did not build - and that I only really touch when something is broken or needs maintenance.

Disclaimer: I am not an infrastructure person by nature. I do this because I need to deploy my code - not because I am overly excited about provisioning tooling. If you are motivated by having disaster recovery plans and cross regional contingencies - this article is probably not for you.

Rough thoughts on other tools I have used

  • Kubernetes: Way overkill for what I needed, and I found it very difficult to remember how to use it as I would only interact with it so rarely. Ended up moving any Kubernetes workflows that I inherited over to ECS.
  • AWS ECS + CDK: I actually don't mind CDK. What I found grating about the experience had a lot more to do with AWS than to do with CDK specifically. We had to build a lot of tooling around deployments ourselves.
  • Terraform + Ansible: I also liked Terraform, because I actually liked the constraints that the language introduced. It wasn't trying to be too smart, which I appreciated. Where it became painful was around provisioning and keeping the servers up to date once they were out in the wild. I used Ansible for this - which worked but always felt a bit clunky.
  • Docker compose: I have some simple servers that I still run that just use docker compose directly on the host - and I have some simple bash scripts to pull and restart containers. Works well enough, but never felt like a proper solution.
  • Heroku: Despite it's flaws, is still actually nice to use. The killer is the SalesForce ownership and its slow death. They are rolling out a redesign which will probably kill it.
  • Docker: A neccessary evil. While it introduces its own problems and complexities, the tradeoffs seem worth it to me.

I am being intentionally vague about the term infrastructure tooling here - these options all do different things in slightly different ways. Overall the task I am trying to solve is deploying a web server. The split between IAC / provisioning / orchestration isn't one I am going to go into here.

Things that were always painful across these tools

  • Coming back to a project after a year and having all the tooling still work
    • Nothing worse than trying to wrestle with Poetry on an out of date CDK cli vs library version mismatch
  • AWS (or GCP or other platform). Despite having worked at Amazon, I am not a huge fan of AWS. Trying to get CORS working (or the correct IAM permissions between task roles and execution roles) often felt like pulling teeth. It is very very easy to rack up expensive bills - it requires constant attention to make sure that people on the team are using it in the "right" way.
  • Complexity
    • It often requires a lot of code and files to do something that feels like it should be simpler. I want to deploy my web server to a machine.
    • Often what would have been 'simple' tasks on regular servers such as running cron jobs have become quite difficult. On one CDK system that I maintain, it required 5+ yaml files and python code changes to be able to just run a cron job. Madness.
  • Secrets management
    • And environment variables
    • I always liked the ways that modern Rails handles this, with the encrypted secrets files. It meant you could commit your secrets, which I like

Kamal

I have been using Kamal for deploying servers recently, and have found a lot to love. Like any infra tool - there are some rough edges.

The core concept is managing deployments of your apps to just plain old ubuntu boxes. The provisioning side can be as simple as click ops-ing a server on DigitalOcean.

The core tech is really two parts:

  • The proxy: Runs on the host server and proxies the requests down to your web servers. This means you can run multiple apps or systems on your single server, and the proxy will route the traffic to the correct place.
  • Docker containers: You give kamal a Docker container to run, and through a series of ruby / shell scripts it will handle anything to do with the lifecycle.

Things I like about Kamal

  • Deployments
    • Very simple and fast deployments. kamal deploy will do a zero downtime deployment of your app.
    • Because of the Docker layer cache, most of the time the builds are pretty rapid and the whole deployment process takes under a minute.
  • Quality of life
    • Simple to spin up services that are not your core web app. This can range from job queues through to running postgres containers.
    • Easy to SSH into the running container, whether it is your app or a postgres container.
  • Certs
    • Simple to chuck a domain in front of your app, with https working without issues
  • It's still a server
    • The host server which is running kamal proxy is just a regular ubuntu box. You can ssh to it directly. You can install things on it if you want. For instance, I have a server where I have installed NVIDIA container toolkit on the host so containers running on it can access CUDA.
    • You can choose your provider. I find that I get way more performance per $ from a DigitalOcean server than ECS. I'm sure that others offer even more (hetzner etc). And you can also use this on your on-prem servers.
  • Secrets
    • Done sensibly, with an integration into 1Password
    • Allows me to commit my secrets file (without actually having the secrets in it)

Kamal by example

This website is a self-hosted Ghost instance. So naturally, I am deploying it using Kamal.

I have a single $7USD/month server on DigitalOcean

The server hosting this very website

There are 3 files that are required to run Kamal

  1. Dockerfile
FROM ghost:5-alpine

Currently that is the extent of my Dockerfile, I am not doing any customisation to the base ghost image.

  1. config/deploy.yml
# Name of my application. Used to uniquely identify this when running multiple kamal deployments on the same server
service: andy-website

# Name of the container image to be pushed during build, and pulled onto the server to deploy
image: andrewsbltech/andy-website

# Deploy to these servers. 170.64.156.161 is my DigitalOcean server
servers:
  web:
    - 170.64.156.161


proxy:
  ssl: true
  # provisions the cert for my domain
  host: andrewperkins.com.au
  # Some config changes to make the proxy understand when ghost is booted
  app_port: 2368
  healthcheck:
    interval: 3
    path: /ghost/api/admin/site
    timeout: 5

# Credentials for image host. I am using DockerHub.
registry:
  username: andrewperkins
  password:
    - KAMAL_REGISTRY_PASSWORD

# Build for amd64
builder:
  arch: amd64

# services that I want to run as well as my ghost instance. Here I need mysql as that is what ghost uses. Runs on the same server.
accessories:
  db:
    image: mysql:8.0
    host: 170.64.156.161
    env:
      secret:
        - MYSQL_ROOT_PASSWORD
    options:
      restart: always
    directories:
      - data:/var/lib/mysql


# Env vars that will be sent through to the ghost container
env:
  clear:
    database__client: mysql
    # note that this database connection is managed via docker networking, it just works when you specify {service}-{accessory}
    database__connection__host: andy-website-db
    database__connection__user: root
    database__connection__database: ghost
    privacy__useUpdateCheck: false
    privacy__useGravatar: false
    url: https://andrewperkins.com.au
    NODE_ENV: production
  secret:
    - database__connection__password

aliases:
  # this allows me to ssh into the running container
  shell: app exec --interactive --reuse "bash"

# persist the ghost content to the host file system
volumes:
  - "ghost:/var/lib/ghost/content"
  1. .kamal/secrets
SECRETS=$(kamal secrets fetch --adapter 1password --account "my" --from "Private/Andy Website" KAMAL_REGISTRY_PASSWORD MYSQL_ROOT_PASSWORD)
KAMAL_REGISTRY_PASSWORD=$(kamal secrets extract KAMAL_REGISTRY_PASSWORD $SECRETS)
MYSQL_ROOT_PASSWORD=$(kamal secrets extract MYSQL_ROOT_PASSWORD $SECRETS)
database__connection__password=$(kamal secrets extract MYSQL_ROOT_PASSWORD $SECRETS)

This is one of my favourite parts of Kamal. This file is designed to be committed (or shown publicly!). I read the secrets out of a 1Password item that contains both a registry password (for storing the Docker image), and a database password for mysql.

Overall

I have been really enjoying Kamal, and plan to keep using it. The ability to deploy to on-prem servers is a big deal, as I manage a number of projects that need this ability.