Write runbooks for the person who just got woken up at 3am

I have been on call for about nine years now across three companies and I can tell you the single biggest thing that decides if an incident takes 12 minutes or 3 hours is not the monitoring stack, its not kubernetes, its not how smart your team is. its whether the runbook was written for a person who is half asleep.

Most runbooks I've inherited were written by someone who was wide awake, had full context, and was kind of showing off. They read like a design doc. Three paragraphs of background on why the service exists, a diagram, then somewhere in paragraph six there's the actual command you need and it has a typo in it.

Here's how we write them now. Took us a couple of quarters to get every paging alert covered but our time to mitigate dropped a lot and more importantly people stopped dreading the pager.

1. First line: what is broken for customers

Not "p99 latency on shipment-api above threshold". Thats the alert name, I can already read that. The first line answers "who is hurting right now". Like: Drivers cannot submit proof of delivery. Dispatchers will start calling within 15 min. That one sentence tells the on-call how fast to move and who to warn.

2. Copy-paste commands, with the real names in them

If the runbook says "check the pod logs" you have failed. Put the exact command. The namespace, the label selector, the context name. Half asleep me will absolutely run it against the wrong cluster otherwise.

kubectl --context prod-us-east -n shipments get pods -l app=pod-upload
kubectl --context prod-us-east -n shipments logs -l app=pod-upload --since=15m | grep -i error | tail -50

Yes its long. Long is fine. Long and correct beats short and "you know what I mean".

3. A decision tree, not an essay

Every runbook has a small "if this then that" section. If error rate is high AND there was a deploy in the last hour, roll back first, ask questions later. If error rate is high and no deploy, check the upstream (with the dashboard name written out). If its only one region, fail over using this exact command. People under stress cant hold five possibilities in their head so dont make them.

4. Mitigate first, fix later, and say so out loud

We literally write "you do not need to find the root cause tonight" at the top of the big ones. I have watched good engineers spend 90 minutes debugging a memory leak at 4am while customers were down, when restarting the deployment would have bought them a whole day. Rollback, restart, scale up, fail over, flag off. Get the bleeding stopped. The postmortem is where you get smart.

5. Who to wake up, and permission to do it

Name the escalation path with the actual rotation names. And give explicit permission: "if this is not mitigated in 30 min, page the database on-call, they will not be mad". Juniors especially will sit alone with an incident for an hour because they dont want to bother anybody. That is a culture problem but the runbook can fix part of it.

6. Last verified date, and a name

Every runbook has "last verified: date, by: person". If its older than 6 months it shows up in a report and someone has to walk through it again. Runbooks rot FAST. Services get renamed, dashboards move, the cluster context changes after a migration and suddenly your perfect command points at nothing.

How we keep them honest

Once a month we do a small game day. Someone picks a random alert, breaks the thing in staging, and a person who did NOT write the runbook has to follow it, out loud, on a call. No help allowed for the first 10 minutes. Its humbling. Every single time we find a missing step, a permission the on-call doesnt have, or a command that needs a flag nobody wrote down.

The other rule: if an alert fires and the runbook didnt help, fixing the runbook is part of closing the incident. Not a follow up ticket that dies in the backlog. Part of closing it.

And delete alerts that have no runbook

If you cant write what a human should DO when the alert fires, then it shouldnt page a human. Make it a dashboard or a ticket. We deleted a bunch of alerts this way and nobody missed them, including the people who originally created them.

Anyway. Write it for tired you. Tired you is not as smart as you think and tired you will be very grateful.