Our worst week this year: a queue job that charged one customer three times

I post the wins here, so I should post this one too.

UK ecommerce client, Laravel, a payment gateway the client chose (a smaller provider, not Stripe). Checkout puts a ChargeOrder job on the queue. The job had $tries = 3 with backoff, because "network errors happen". One evening the gateway was slow. The charge request took longer than our HTTP timeout, so the job threw a timeout exception. Laravel did exactly what we told it: retried. Twice.

The gateway had not failed. It was just slow. All three charges went through. £189, three times, on one customer's card. She found out from her banking app before we found out from our logs.

What we got wrong

• We treated a timeout as a failure. A timeout means "I don't know". The money might have moved.
• We retried a non-idempotent call to the outside world. This gateway has no idempotency key support, so every retry was a new charge.
• Nothing in our database said "a charge is in flight for this order", so every attempt looked like the first.

The fix

class ChargeOrder implements ShouldQueue
{
    use Queueable;

    public int $tries = 1; // never retry a charge blindly

    public function __construct(public int $orderId) {}

    public function handle(Gateway $gateway): void
    {
        $mine = DB::transaction(function () {
            $order = Order::lockForUpdate()->findOrFail($this->orderId);

            if ($order->payment_state !== PaymentState::Pending) {
                return false; // someone already tried, reconcile decides
            }

            $order->update(['payment_state' => PaymentState::Charging]);

            return true;
        });

        if (! $mine) {
            return;
        }

        try {
            $txn = $gateway->charge($this->orderId, reference: "order-{$this->orderId}");

            Order::whereKey($this->orderId)->update([
                'payment_state' => PaymentState::Paid,
                'gateway_txn' => $txn->id,
            ]);
        } catch (GatewayTimeout $e) {
            // we do NOT know if it charged. ask the gateway later, by our reference.
            ReconcilePayment::dispatch($this->orderId)->delay(now()->addMinutes(2));
        }
    }
}

The order moves Pending → Charging → Paid, and only a job that wins the row lock while the order is still Pending is allowed to call the gateway. On a timeout, a separate ReconcilePayment job asks the gateway for the transaction by our reference and only then decides: mark it paid, or put it back to Pending so the customer can try again.

If your gateway supports idempotency keys (Stripe does), pass one. With Stripe's PHP library it is one extra array: ['idempotency_key' => "charge-order-{$order->id}"]. Retries become safe. Ours does not, so we built the state machine. Should have built it on day one anyway.

We refunded the customer the same evening, the client kept us (alhamdulillah), and I re-read emeka_golang's idempotency post from July with a lot of regret. Timeouts are "I don't know". Write it on the wall.