How a Missing HTTP Timeout Let a Dead Webhook Endpoint Quietly Eat Our Entire Job Worker Pool

04:58 UTC. The on-call phone lights up with a single alert: webhook delivery queue depth has crossed 40,000 and is climbing in a straight line. No deploy in the last eighteen hours. No traffic spike. Worker CPU graphs are flat and low, like the pool is barely doing anything, which is strange for a queue that's supposedly backing up this fast.

the scramble

First theory: a worker crash loop. The orchestrator dashboard says otherwise, all 24 worker processes show as running, no restarts in the last six hours. Second theory: Postgres itself is slow, maybe a bad query plan after an autovacuum ran long. A quick check of pg_stat_activity shows nothing unusual on CPU or I/O wait. Third theory, the standard one for "queue won't drain": something is holding a lock that everything else is stuck behind.

 pid | wait_event_type | wait_event | query
-----+------------------+------------+-------
(0 rows)

Zero rows. Nobody is waiting on a lock. That's the first dead end, and it's a convincing one, because this is the exact query every Postgres incident runbook tells you to run first. It comes back clean because the jobs table was deliberately designed to never show contention here.

the hunt

The job queue is a plain Postgres table, jobs(id, tenant_id, webhook_url, payload, status, created_at), and every worker claims the next row with the same query:

BEGIN;

SELECT id, tenant_id, webhook_url, payload
FROM jobs
WHERE status = 'pending'
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 1;

SKIP LOCKED is the whole point of this design: if another worker already has a row locked, you skip it and grab the next one instead of blocking. That's why the lock-wait query above returns nothing, nobody blocks on anything by construction. But a lock that's held for five hours and never contended for also never shows up anywhere a dashboard would look. Checking pg_locks directly for long-held grants, not waiters, tells a different story:

 pid  | state  | held_for | query
------+--------+----------+------------------------------------------
 9142 | active | 05:11:38 | SELECT id, tenant_id, webhook_url ...
 9188 | active | 04:47:02 | SELECT id, tenant_id, webhook_url ...
 9201 | active | 04:22:51 | SELECT id, tenant_id, webhook_url ...
 9340 | active | 03:58:14 | SELECT id, tenant_id, webhook_url ...

Four backends, all showing the same claim query, all stuck for hours, all with state = active rather than idle, meaning each one is still waiting on something inside the transaction, not sitting idle after it. Pulling the application logs for those four PIDs shows every one of them in the exact same place: a requests.post() call to the same webhook_url, for the same tenant_id. One customer's integration endpoint stopped responding at some point the previous evening, no TCP reset, no connection close, just silence after the handshake completed, almost certainly a misconfigured firewall on their side dropping the response rather than rejecting the request outright. The worker's HTTP client had never been given a timeout, so the call simply never returned.

That explains one stuck worker. It does not explain a pool-wide outage, because SKIP LOCKED means a stuck row costs you exactly one worker, not the whole queue. The second part of the picture comes from the jobs table itself: this customer's integration retries failed deliveries by enqueuing a fresh row, and new events for that tenant kept arriving all night at a normal rate. Every one of those new jobs got claimed successfully, by a different, healthy worker, because SKIP LOCKED let them skip right past the already-stuck rows. Then each of those workers made the same requests.post() call to the same dead endpoint, with the same missing timeout, and got stuck too.

the find

A single customer's webhook endpoint went silent without closing the connection, and the worker code held a database transaction open across an HTTP call with no timeout. SKIP LOCKED correctly prevented that from blocking any other job, which is exactly why nobody could see it happening: each new job for that tenant recruited one more worker into the same hang, one at a time, over roughly five hours, until all 24 workers in the fixed-size pool were permanently stuck on the same dead endpoint. At that point there were zero workers left to process jobs for any tenant, and the queue depth alert that fired at 04:58 UTC was the first moment the damage became visible outside this one customer's own traffic.

the fix

The immediate fix was blunt: restart the entire worker pool. Killing the processes drops the hung sockets and the open Postgres transactions with them, which clears every stuck row's lock at once.

deployment.apps/webhook-workers restarted

Queue depth started dropping within two minutes, and the backlog that had built up across every tenant drained in about eleven minutes once a full pool of healthy workers was processing again. The real fix had two parts. First, every outbound call gets an explicit timeout, no exceptions:

session = requests.Session()
response = session.post(
    job.webhook_url,
    json=job.payload,
    timeout=(3, 10),  # connect, read
)

Second, and more important: the HTTP call no longer happens inside the claiming transaction at all. A worker now claims a job and commits immediately, marking it in_progress with a lease timestamp, does the HTTP call outside any open transaction, and only opens a second, short transaction afterward to record success or failure:

with conn.transaction():
    job = claim_and_mark_in_progress(conn)  # commits here, no lock held after

if job:
    try:
        response = session.post(job.webhook_url, json=job.payload, timeout=(3, 10))
        outcome = 'done' if response.ok else 'failed'
    except requests.RequestException:
        outcome = 'failed'

    with conn.transaction():
        finalize_job(conn, job.id, outcome)

Even a client that still somehow hangs forever now only holds a worker process, never a database row lock or an open transaction, and a reaper job reclaims anything stuck in in_progress past a 60-second lease.

the aftermath

No jobs were lost, every pending delivery eventually went out once the pool restarted, but webhook delivery for every tenant, not just the one with the broken endpoint, stalled for roughly fifty minutes while the on-call team chased the wrong diagnostic first.

timeout=(3, 10) would have stopped this in its first minute. The dangerous part wasn't the missing timeout on its own, it was pairing that gap with a claim pattern that let one slow call hold a database transaction open for as long as the network stayed quiet, and a queueing primitive specifically built to make that invisible until it had already taken down the whole pool.