I took our UPI collect flow down for 14 minutes on a Friday evening

Every Friday deploy story starts with "it was just a config change". So here is mine.

Our partner bank rotates the public key they use to sign payment callbacks once a year. They emailed us the new key, I updated the config and deployed at 6pm on Friday because the old key was expiring on Saturday. Felt responsible. Was not.

We run the callback service on 2 pods. The config came from a mounted secret, one pod picked up the new value and the other one was still on the old value because it had not restarted. So half the callbacks were rejected with signature errors. Payments stayed in pending, users saw "processing" forever, and many of them retried, which created new collect requests on top of the stuck ones.

We noticed after 9 minutes from the pending dashboard, not from an error alert, because rejected callbacks were logged as warnings. Rolled back the config, restarted both pods, then ran our status poller manually to fix the stuck ones. 14 minutes total. Around 600 payments affected, no money lost, lot of confused users and one very long Saturday of reconciliation.

What we changed after:

• Signature verification accepts a list of keys, so during rotation both old and new are valid. Remove the old one a week later.
• Rejected callbacks are errors with an alert, not warnings.
• Alert on ratio of payments in pending older than 5 minutes. This is the metric that actually tells you users are stuck.
• Secret changes trigger a rolling restart of every pod, not "whenever".

And yes, no deploys after 4pm Friday. My manager did not even have to say it.