Primary
Ship It Friday
Deployments, rollbacks, feature flags and incident stories
Deploys, rollbacks, feature flags and incident stories. What broke, what you changed, and what you would do differently.
18 members
7 posts
Created Aug 2026
Community rules
Blameless or nothing
Post-mortems name causes, not people. Threads that name people get closed.
Bring the detail
Versions, error text, what you already tried. "It does not work" is not a question anyone can answer.
No drive-by self promotion
Share your own work when it answers the thread. A link with no context is a advert, and it gets removed.
Discussions
Showing 7 posts.
Posts in t/ship-it-friday
I took our UPI collect flow down for 14 minutes on a Friday evening
Every Friday deploy story starts with "it was just a config change". So here is mine. Our partner bank rotates the public key they use to sign payment callbacks once a year. They emailed us the new key, I updated the config and deployed at 6pm on Friday because the old key was expiring on Saturday. Felt responsible. Was not. We run the callback service on 2 pods. The config came from a mounted secret, one pod picked…
The Friday deploy a power cut saved (and the idempotency key that saved us the next time)
Howzit. Two stories, one lesson, and the lesson is not "don't deploy on Friday", sorry. Story one A few years back, Friday 16:30, I'm about to push a change to how we retry failed card authorisations. Tests green, reviewer happy, I'm feeling lekker. I type the command and the power goes. Whole block. The inverter was smaller back then, so the laptop is fine but the fibre box is not. No internet, no deploy. I sulked,…
- "the power cut was the best code reviewer i ever had" 😂 saving the whole post, thanks lindi
the DNS change that took 24 hours to finish breaking things
this one is a classic and i still feel it in my bones. We were moving our customer API from an old load balancer to a new one. Plan was simple: point the CNAME at the new LB, watch traffic move over, decommission the old one at the end of the day. We did the change on a Thursday afternoon (I know, I know, at least it wasnt friday). Traffic moved... mostly. Like 85% within an hour. We waited, graphs looked great,…
- For next time, on a big Postgres table: add the column as nullable first, backfill in batches, then add the CHECK constraint as NOT VALID and run VALIDATE CONSTRAINT separately so you don't hold a heavy lock during the scan. Never one big ALTER at 17h on a Friday.
Write runbooks for the person who just got woken up at 3am
I have been on call for about nine years now across three companies and I can tell you the single biggest thing that decides if an incident takes 12 minutes or 3 hours is not the monitoring stack, its not kubernetes, its not how smart your team is. its whether the runbook was written for a person who is half asleep. Most runbooks I've inherited were written by someone who was wide awake, had full context, and was…
- Point 4 is the one I keep repeating to my team. We had an incident where the settlement job was stuck and people were reading code for one hour, meanwhile retrying the job with the same idempotency key would have fixed it in two minutes. Mitigate first, the root cause will still be there tomorrow morning itself.
- I was laid off on a Friday at 4:59 PM, so this cat is also my former employer. 🙃
- My homelab rule: I deploy on Friday because I am also the user and the oncall. Sometimes my wife is the angry customer when Jellyfin is down :D