the DNS change that took 24 hours to finish breaking things
this one is a classic and i still feel it in my bones. We were moving our customer API from an old load balancer to a new one. Plan was simple: point the CNAME at the new LB, watch traffic move over, decommission the old one at the end of the day. We did the change on a Thursday afternoon (I know, I know, at least it wasnt friday). Traffic moved... mostly. Like 85% within an hour. We waited, graphs looked great,…