the DNS change that took 24 hours to finish breaking things

this one is a classic and i still feel it in my bones.

We were moving our customer API from an old load balancer to a new one. Plan was simple: point the CNAME at the new LB, watch traffic move over, decommission the old one at the end of the day. We did the change on a Thursday afternoon (I know, I know, at least it wasnt friday).

Traffic moved... mostly. Like 85% within an hour. We waited, graphs looked great, someone said "the long tail is just slow clients" and at 6pm we turned off the old load balancer.

Next morning the support queue was on fire. A big chunk of our enterprise customers integrations were failing. All of them sat behind corporate resolvers that had cached the OLD record, and the old record had a TTL of 86400. Twenty four hours. Somebody set that years ago "for performance" and nobody ever looked at it again. We had lowered the TTL to 60 seconds... about one hour before the change. Which does nothing for a resolver that cached the old 24h answer the day before.

What we do now, every time:

• Lower the TTL at least one full old-TTL period before the change. If it was 86400, lower it two days before.
• Keep the old target alive and healthy for at least the old TTL after the switch. Watch its traffic go to zero before you kill it, dont assume.
• Turn on access logs on the old target so you can see WHO is still hitting it, and call them if needed.
• Some clients cache DNS inside the process way longer than the TTL (some old JVM configs cache forever). For those you need a restart on their side so tell partners in advance.

The fix took 2 minutes, turning the old LB back on. The trust with those customers took a lot longer.