reminder that "its a one line change" is how every…
reminder that "its a one line change" is how every postmortem i have ever written starts. deploy on friday if you want, im not your dad. but i am the one getting paged
#incidents
Posts tagged #incidents on Dynamo Geeks: 4 posts from the community, most useful first.
Posts tagged #incidents
I took our UPI collect flow down for 14 minutes on a Friday evening
Every Friday deploy story starts with "it was just a config change". So here is mine. Our partner bank rotates the public key they use to sign payment callbacks once a year. They emailed us the new key, I updated the config and deployed at 6pm on Friday because the old key was expiring on Saturday. Felt responsible. Was not. We run the callback service on 2 pods. The config came from a mounted secret, one pod picked…
the DNS change that took 24 hours to finish breaking things
this one is a classic and i still feel it in my bones. We were moving our customer API from an old load balancer to a new one. Plan was simple: point the CNAME at the new LB, watch traffic move over, decommission the old one at the end of the day. We did the change on a Thursday afternoon (I know, I know, at least it wasnt friday). Traffic moved... mostly. Like 85% within an hour. We waited, graphs looked great,…
Write runbooks for the person who just got woken up at 3am
I have been on call for about nine years now across three companies and I can tell you the single biggest thing that decides if an incident takes 12 minutes or 3 hours is not the monitoring stack, its not kubernetes, its not how smart your team is. its whether the runbook was written for a person who is half asleep. Most runbooks I've inherited were written by someone who was wide awake, had full context, and was…