Ship It Friday: top posts
The most useful posts in this topic, by likes from the community.
Posts
The Friday deploy a power cut saved (and the idempotency key that saved us the next time)
Howzit. Two stories, one lesson, and the lesson is not "don't deploy on Friday", sorry. Story one A few years back, Friday 16:30, I'm about to push a change to how we retry failed card authorisations. Tests green, reviewer happy, I'm feeling lekker. I type the command and the power goes. Whole block. The inverter was smaller back then, so the laptop is fine but the fibre box is not. No internet, no deploy. I sulked,…
Since 3 years I ask my team to not deploy on friday. Today…
Since 3 years I ask my team to not deploy on friday. Today is friday, 17h. Guess what we deploy ? A "small migration" on the users table. Putain. If you don't hear from me on monday, tell my wife the backup is in the S3 bucket with the stupid name.
I took our UPI collect flow down for 14 minutes on a Friday evening
Every Friday deploy story starts with "it was just a config change". So here is mine. Our partner bank rotates the public key they use to sign payment callbacks once a year. They emailed us the new key, I updated the config and deployed at 6pm on Friday because the old key was expiring on Saturday. Felt responsible. Was not. We run the callback service on 2 pods. The config came from a mounted secret, one pod picked…
Write runbooks for the person who just got woken up at 3am
I have been on call for about nine years now across three companies and I can tell you the single biggest thing that decides if an incident takes 12 minutes or 3 hours is not the monitoring stack, its not kubernetes, its not how smart your team is. its whether the runbook was written for a person who is half asleep. Most runbooks I've inherited were written by someone who was wide awake, had full context, and was…
how does your team actually do on-call?
the DNS change that took 24 hours to finish breaking things
this one is a classic and i still feel it in my bones. We were moving our customer API from an old load balancer to a new one. Plan was simple: point the CNAME at the new LB, watch traffic move over, decommission the old one at the end of the day. We did the change on a Thursday afternoon (I know, I know, at least it wasnt friday). Traffic moved... mostly. Like 85% within an hour. We waited, graphs looked great,…