Idempotency keys for payment APIs: what we got wrong the first time

Last year we had a customer charged twice for the same EMI. Not a big amount, but the customer was right to be angry and our support team spent one full day itself on it. The root cause was not the bank, not the gateway. It was us, and the way we did idempotency keys.

Sharing what we got wrong, because since then I have seen the same mistakes in two other codebases.

What an idempotency key is supposed to do

The client sends a unique key with any request that has side effects, like "charge this card". If the same key comes again, the server does not do the work again, it returns the result of the first attempt. So the client can retry safely on a timeout or a flaky network, which on mobile in India is basically every request.

Mistake 1: generating the key per HTTP attempt

Our app generated a fresh UUID inside the HTTP client, so every retry had a new key. Which means we had idempotency on paper and zero idempotency in practice. The key must represent the user's intent, one tap on "Pay", and it must be created before the first attempt and reused for every retry of that intent. If the user leaves the screen and comes back and taps again, that is a new intent and a new key.

Mistake 2: saving the key after calling the gateway

Our flow was: call gateway, get response, then insert key and response into DB. If the process crashed between the gateway call and the insert (it did, during a deploy), the retry found no key and charged again. Now we do it in this order:

1. Insert the key row with status started and a hash of the request body. Primary key on (merchant_id, key).
2. If the insert conflicts, read the existing row. If it is finished, return the stored response. If it is still started, return 409 and let the client retry after some time.
3. Call the gateway, and pass a downstream idempotency key derived from our key. Most gateways support this, read their docs.
4. Store response code and body on the row, mark it finished.

create table idempotency_keys (
  merchant_id   bigint not null,
  key           text not null,
  request_hash  text not null,
  status        text not null default 'started',
  response_code int,
  response_body jsonb,
  created_at    timestamptz not null default now(),
  primary key (merchant_id, key)
);

Step 3 is the important one. If we crash after the gateway call, our retry sends the same downstream key, so the gateway itself deduplicates. Two layers, and each one covers the other one's gap.

Mistake 3: same key, different body

A bug in the app once reused a key for a different amount. Our server happily returned the old response for the new request, so the user saw "success" for a payment that never happened at that amount. Now we hash the request body and if the hash does not match we return 422. Same key with a different payload is a client bug and it should be loud.

Mistake 4: expiry too short

We deleted keys after 1 hour to keep the table small. Our Android app has an offline queue that can retry hours later when the phone gets network again. Keep keys at least as long as your longest client retry window. We now keep them 7 days and partition by day so dropping old ones is cheap.

Small things that also matter

• Scope the key per merchant or per user. A global namespace means one client's key can collide with another's.
• Do not log the full request body along with the key. You will find card data in your log search one day.
• Timeouts from the gateway are not failures. Mark them unknown and check the status before telling the user anything.

None of this is new, payment companies have written about it for years. But reading about it and having it actually work under retries and crashes are different things. Test it by killing the process in between the steps, thats the only test that convinced me.