The Friday deploy a power cut saved (and the idempotency key that saved us the next time)

Howzit. Two stories, one lesson, and the lesson is not "don't deploy on Friday", sorry.

Story one

A few years back, Friday 16:30, I'm about to push a change to how we retry failed card authorisations. Tests green, reviewer happy, I'm feeling lekker. I type the command and the power goes. Whole block. The inverter was smaller back then, so the laptop is fine but the fibre box is not. No internet, no deploy. I sulked, went home, braaied, slept.

Monday morning I look at the change with fresh eyes and realise the retry loop would have retried authorisations that had actually succeeded at the processor but timed out on our side. In payments that's called charging people twice. The power cut was the best code reviewer I have ever had.

Story two

The real fix wasn't "pray for load shedding". The bug was that our retries weren't idempotent. A request that timed out on our side might have succeeded downstream and we had no way to know, so every retry was a coin toss.

What we do now, and what I'd tell anyone touching money:

• Every mutating request gets an idempotency key, generated once by the client when the user presses the button, not per attempt. Retries reuse the same key.
• The server stores the key with the result before it returns. Same key again? Return the stored result, don't do the work twice.
• Uniqueness lives in the database, not in application code, so two retries racing each other can't both win:

CREATE TABLE idempotency_keys (
  key          text PRIMARY KEY,
  request_hash text NOT NULL,
  response     jsonb,
  created_at   timestamptz NOT NULL DEFAULT now()
);

-- first writer wins, everyone else reads the stored response
INSERT INTO idempotency_keys (key, request_hash)
VALUES ($1, $2)
ON CONFLICT (key) DO NOTHING;

• Check the request hash. Same key with a different body is a client bug. Return a 422 and shout about it in the logs.
• Assume the lights go out mid-transaction. Because they will, literally or because a pod gets killed halfway. Every step should be safe to run twice.

Stripe's API has worked like this for years with its Idempotency-Key header, so it's not some exotic pattern. It just isn't the default in most codebases until somebody gets charged twice.

So, Friday deploys? Ja, we do them. Small, behind a flag, with a rollback we've actually tested. But the rule I care about is: if running it twice would hurt someone, it's not ready, Friday or Monday.

Shame, the inverter is bigger now. I've lost my best reviewer.