It started with my dad explaining what Kubernetes was. Once I understood the idea of running software across a group of machines instead of on one, we looked at the different ways to set it up and decided on K3s, a lightweight Kubernetes distribution, instead of full upstream Kubernetes or the other options.

The first deploy went onto a small Lenovo ThinkCentre I had already been using to run Docker. Moving from a handful of containers on one box to an actual cluster was where I learned how Kubernetes works in practice: how workloads get scheduled, how services find each other, and what happens when something breaks.

It didn't stay small for long. I started buying used Dell PowerEdge servers from Garland Computer, and when Christmas came around I asked for more of them. Today the cluster is eight nodes, two control-plane nodes and six workers, made up of two R630s, five R730xds and a Supermicro GPU server running eight AMD Instinct MI50s. The R730xds hold over 30 TB of storage on SAS drives.

Every node runs NixOS. Instead of setting up each server by hand, all of them are built from one standardized Nix configuration. When a new server arrives, I install NixOS on it over SSH with nixos-anywhere, and from then on I use Colmena to roll the configuration out to groups of machines at once. A new node comes up the same way as the rest, and a change ships to the whole fleet with one command.