Posts tagged #distributed-systems on Dynamo Geeks: 4 posts from the community, most useful first.
timeouts are not failures. if the bank API times out you dont know whether money moved. mark it UNKNOWN, then query status or retry with the same idempotency key. never show 'payment failed, try again' on a timeout, thats how double charges happen
Every payment gateway webhook doc says some version of "we may send the same event more than once". What they say less loudly: events can arrive out of order, and sometimes they dont arrive at all. Our first webhook handler assumed none of this. Here is the version that survived. 1. Twice: store the event id Every event has an id. Insert it into a processed_events table with a unique constraint before doing anything…
We have a payouts table and 4 worker processes that pick pending payouts and call the bank API. Right now each worker does this: begin; select * from payouts where status = 'pending' order by created_at limit 10 for update skip locked; -- call bank API for each row (can take up to 30s) update payouts set status = 'sent' where id = ...; commit; It works, no double picks. But the transaction stays open during the HTTP…
Last year we had a customer charged twice for the same EMI. Not a big amount, but the customer was right to be angry and our support team spent one full day itself on it. The root cause was not the bank, not the gateway. It was us, and the way we did idempotency keys. Sharing what we got wrong, because since then I have seen the same mistakes in two other codebases. What an idempotency key is supposed to do The…
