Payment webhooks arrive twice, out of order, and sometimes never. Design for all three

Every payment gateway webhook doc says some version of "we may send the same event more than once". What they say less loudly: events can arrive out of order, and sometimes they dont arrive at all. Our first webhook handler assumed none of this. Here is the version that survived.

1. Twice: store the event id

Every event has an id. Insert it into a processed_events table with a unique constraint before doing anything else. If the insert fails on conflict, return 200 and stop. Return 200, not 409, otherwise the gateway keeps retrying something you already handled.

2. Acknowledge fast, process later

Verify the signature, store the raw event, return 200. Then process from a queue. Gateways have short timeouts, often a few seconds, and if your handler does DB updates, sends emails and calls three services inline, one slow day will make the gateway retry everything and you are back to problem 1, now with extra load.

3. Out of order: state machine, not last write wins

We once got payment.captured and then, 2 seconds later, an older payment.authorized that was being retried. Our handler set the status back to authorized. The order looked unpaid and the customer got a "complete your payment" email for something they had paid. Fix: payment status is a state machine with allowed transitions. Authorized can go to captured, captured can never go back to authorized. An event that would move backwards is logged and ignored.

const allowed = {
  created:    ['authorized', 'failed'],
  authorized: ['captured', 'failed', 'voided'],
  captured:   ['refunded', 'partially_refunded'],
}

Even better, when an event arrives, fetch the current object from the gateway API and use that as the truth. The event then is just a notification that "something changed", which makes the order irrelevant. Costs one API call per event, worth it for payments.

4. Never: poll for the ones that went missing

Webhooks fail. Your endpoint was down during a deploy, a firewall rule changed, the gateway had an incident. Run a job every 10-15 minutes that finds payments still pending after N minutes and asks the gateway for their status. This job saved us more than once when webhooks silently stopped for a few hours.

5. Verify the signature on the raw body

Compute the HMAC on the exact raw bytes you received, not on JSON you parsed and serialized again. Key order and whitespace change, the signature fails, someone "temporarily" disables verification, and it stays disabled for a year. Compare with a constant time function and reject old timestamps if the gateway sends one.

Most of this is boring plumbing. But each point is a production incident we actually had, so maybe it saves you one.