Four Pi 5s, one k3s cluster: full build log, what went wrong, and why I'd buy a mini PC next time

G'day legends. This took three weekends, two trips to Jaycar and one very long conversation with the wife about "what exactly is a control plane". Years ago Jeff Geerling's Pi Dramble videos put the idea in my head and it never left, so if you've not seen the original, start here:

This is the full log. Every command I ran, everything that went wrong, and at the end an honest answer to "should you build one".

Parts list

• 4 × Raspberry Pi 5, 8GB
• 4 × official Active Cooler
• 4 × M.2 HAT+ with a 256GB NVMe each
• 4 × official 27W USB-C power supply (more on why below)
• 1 × 5 port gigabit switch, unmanaged, nothing fancy
• 4 × short Cat6 patch leads, a 120mm USB fan, and a rack I printed in PETG

All up just under AU$1,100, which is a number I have not said out loud in the house.

Step 1: flash and name them

Raspberry Pi OS Lite, 64-bit. I flashed each NVMe directly from my laptop in a USB M.2 enclosure using Raspberry Pi Imager, which is faster than any clever cloning. In Imager's settings I set the hostname (node1 to node4), a user, my SSH public key, and turned off password login. Then DHCP reservations on the router so they're always 192.168.1.51 to .54.

Step 2: boot from NVMe

With the official HAT+ the PCIe port switches on by itself. On the cheap third party HAT I tried first I had to add this to /boot/firmware/config.txt:

dtparam=pciex1
# optional, not officially supported, works on mine, don't blame me:
dtparam=pciex1_gen=3

Then set the boot order so it tries NVMe first:

sudo rpi-eeprom-config --edit
# set:
BOOT_ORDER=0xf416

Read it right to left: 6 is NVMe, then 1 is SD, then 4 is USB, then f means start over.

Step 3: turn on the memory cgroup

This is the one that gets everybody. Pi OS ships with the memory cgroup disabled and k3s refuses to start without it. The log tells you exactly what to do, which is nice of it:

Failed to find memory cgroup, you may need to add "cgroup_memory=1 cgroup_enable=memory" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)

On Bookworm the file is /boot/firmware/cmdline.txt. It's ONE line. Add this to the end of that line, don't put it on a new line:

cgroup_memory=1 cgroup_enable=memory

Reboot all four.

Step 4: install k3s

On node1, the server:

curl -sfL https://get.k3s.io | sh -s - server \
  --write-kubeconfig-mode 644 \
  --disable servicelb \
  --node-ip 192.168.1.51

sudo cat /var/lib/rancher/k3s/server/node-token

On node2, node3 and node4, the agents, using that token:

curl -sfL https://get.k3s.io | \
  K3S_URL=https://192.168.1.51:6443 \
  K3S_TOKEN=K10...the-long-token... \
  sh -

Back on node1:

$ kubectl get nodes
NAME    STATUS   ROLES                  AGE
node1   Ready    control-plane,master   41m
node2   Ready    <none>                 6m
node3   Ready    <none>                 5m
node4   Ready    <none>                 5m

(I trimmed the version column, it'll be newer by the time you read this.) To drive it from the laptop:

scp node1:/etc/rancher/k3s/k3s.yaml ~/.kube/config
sed -i 's/127.0.0.1/192.168.1.51/' ~/.kube/config

Step 5: MetalLB for real IPs

k3s comes with its own little load balancer (ServiceLB, the old klipper-lb) and honestly it's fine. I disabled it because I wanted every service to get its own LAN IP, so Pi-hole can be .240 and Grafana .241 and so on.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml

Then tell it which addresses it may hand out (keep these outside your router's DHCP range!):

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lan
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.240-192.168.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lan
  namespace: metallb-system

Step 6: storage

k3s ships the local-path provisioner, so a PersistentVolumeClaim just works, but the volume lives on one node and the pod is stuck to that node forever. I tried Longhorn for replicated storage. It needs open-iscsi on every node and it felt heavy on the Pis, so I backed it out. Anything that needs to move between nodes now uses an NFS share off the NAS. Not elegant, works.

What went wrong

1. Power (the big one)

First attempt I used a single 100W four port USB-C charger to power all four. Tidy! Wrong. The Pi 5 wants 5V at 5A. On a charger that only offers 3A per port it limits USB current and tells you so at boot ("This power supply is not capable of supplying 5A"). Under load node3 kept browning out:

$ vcgencmd get_throttled
throttled=0x50005

That's under-voltage right now, throttled right now, and both have happened since boot. Four official 27W bricks on a power board later: throttled=0x0 on every node for six weeks. Ugly, but zero throttling.

2. Thermals

In my lovely printed rack the coolers on node1 and node2 faced each other, so node1 blew its hot air straight into node2's intake. node2 sat at 81°C under load while the others were in the 60s. Turned all four the same way and put a 120mm USB fan behind the rack. Max under stress-ng now is 58°C.

3. HA etcd

One server uses SQLite under the hood and that's fine. Being me, I then rebuilt with three servers and embedded etcd (--cluster-init on the first one) for "high availability". On NVMe it's alright. On my test run with SD cards etcd spammed "apply request took too long" and leader elections all night. If you're on SD cards, do not do HA etcd. Just don't.

4. arm64 images

Half the random Docker images I wanted were amd64 only. Before you get attached to one, check:

docker manifest inspect some/image:tag | grep architecture

Power numbers (wall meter, all four nodes plus switch plus fan)

• Idle: 19W
• All four flat out with stress-ng: 46W
• Average over a week with my actual stuff running: 22W

22W around the clock is about 16 kWh a month. At roughly 33c a kWh that's a bit over five bucks a month. Cheaper than I expected.

What actually runs on it

Pi-hole (on its own MetalLB IP), Uptime Kuma, Gitea, Mosquitto for the weather station, and Prometheus plus Grafana. Prometheus retention is set to 7 days because it'll happily eat a whole NVMe.

Would I build it again?

For learning Kubernetes: absolutely. Nothing teaches you how a cluster behaves like yanking a power cable out on purpose and watching what happens. I understand pods, scheduling and node failure properly now in a way I never did from videos.

For running services? Honestly, no. A second hand mini PC with an 8th gen i5 costs less than half of this, idles at about the same wattage, runs every amd64 image on earth, and one box can't have a network partition with itself. If you want to learn k8s, build the Pi cluster. If you want Jellyfin, buy the mini PC. Both are fine answers, just know which question you're asking.

Next up: pulling node2's plug on purpose and timing how long things take to come back. Spoiler, longer than you'd think. A node goes NotReady after about 40 seconds, but by default the pods on it aren't evicted for five minutes.

cheers, Matt 🍓