Skip to content
Payout worker queue in Postgres: SKIP LOCKED is holding transactions open for 30s, better way?
question1 min read95 viewsSep 25, 2026

Payout worker queue in Postgres: SKIP LOCKED is holding transactions open for 30s, better way?

Arjun Mehta@arjunmehta

September 25, 2026 · 1 min read

#1 in SELECT * Anonymous

95 views3 comments

We have a payouts table and 4 worker processes that pick pending payouts and call the bank API. Right now each worker does this:

begin;
select * from payouts
 where status = 'pending'
 order by created_at
 limit 10
 for update skip locked;
-- call bank API for each row (can take up to 30s)
update payouts set status = 'sent' where id = ...;
commit;

It works, no double picks. But the transaction stays open during the HTTP call, and under load we see connections piling up and "idle in transaction" alerts. We are behind PgBouncer.

Options I am thinking:

  • pg_advisory_lock per payout id instead of the row lock
  • a status column with some kind of lease/timeout

Has anyone done this at decent volume (we do ~50k payouts a day)? What is the cleanest pattern that doesnt risk sending the same payout twice?

Accepted answer

Tomasz Kowalski#3 in SELECT * Anonymous4 likes

Do not keep transaction open during HTTP call. Claim, commit, then call. update payouts set status = 'processing', worker_id = $1, lease_until = now() + interval '2 minutes' where id in ( select id from payouts where status = 'pending' order by created_at limit 10 for update skip locked ) returning *; Commit immediately. Then call bank with idempotency key = payout id. Then update to sent or failed. Separate reaper job moves "processing" with expired lease back to pending, but before retry it must ask bank for status, because you dont know if first call reached the bank. Advisory locks: avoid with PgBouncer in transaction pooling. Session level lock lives on server connection which is shared between clients. You will get very interesting bugs.

View in the thread

Written by

Arjun Mehta

Backend engineer at a lending startup in Bengaluru. Payments, ledgers, reconciliation, and the occasional 3am settlement file that does not add up. Postgres fan. Will talk about idempotency keys until you leave.

Get the best of #select-star-anonymous weekly

One email each Monday with the most useful posts. No spam; unsubscribe in one click.

Comments

3

Sign in to join the conversation.

Sign in to comment

Accepted answer

Tomasz Kowalski@tomasz_k#3 in SELECT * Anonymous
Do not keep transaction open during HTTP call. Claim, commit, then call. update payouts set status = 'processing', worker_id = $1, lease_until = now() + interval '2 minutes' where id in ( select id from payouts where status = 'pending' order by created_at limit 10 for update skip locked ) returning *; Commit immediately. Then call bank with idempotency key = payout id. Then update to sent or failed. Separate reaper job moves "processing" with expired lease back to pending, but before retry it must ask bank for status, because you dont know if first call reached the bank. Advisory locks: avoid with PgBouncer in transaction pooling. Session level lock lives on server connection which is shared between clients. You will get very interesting bugs.
4 net votes
Arjun Mehta@arjunmehta#1 in SELECT * Anonymous
This is exactly what I was missing. The pgbouncer point I didnt even think about, we are on transaction pooling. Going with the lease, thanks a lot
1 net votes
Marcus Johnson@marcus_oncall#6 in SELECT * Anonymous
+1 on the lease. also alert on the count of "processing" older than the lease. the reaper quietly fixing things is nice until its re-queuing the same 300 payouts every 2 minutes because the bank is down and nobody knows
1 net votes