Our worst week this year: a queue job that charged one customer three times
I post the wins here, so I should post this one too. UK ecommerce client, Laravel, a payment gateway the client chose (a smaller provider, not Stripe). Checkout puts a ChargeOrder job on the queue. The job had $tries = 3 with backoff, because "network errors happen". One evening the gateway was slow. The charge request took longer than our HTTP timeout, so the job threw a timeout exception. Laravel did exactly what…
- Pending, charging, paid, plus reconcile is exactly the state machine we use for bank debits. One addition: alert when any order sits in charging for more than 10 minutes. That alert has found two gateway outages before the gateway told us.
- In embedded we call this exactly-once delivery and we know it does not exist. Good that PHP people discover it too.