How FastAPI's Default Thread Limiter Let One Slow Endpoint Fail Liveness Probes on Healthy Pods

03:58 UTC. PagerDuty fires on "Liveness probe failures, pod-api, restart count climbing." Within six minutes, eleven API pods have been killed and replaced by kubelet. None of them were unhealthy before the probe said so.

the scramble

On-call pulls up the deploy log first, since a bad rollout is the most common reason pods start failing probes in a tight window. Nothing shipped in the last eight hours. Whatever this is, it isn't a release.

Second theory, and the one everyone expects going in: someone blocked the event loop. It's the textbook FastAPI mistake, a synchronous call sitting inside an async def route, freezing every other request on that worker until it returns. The team has a Prometheus histogram for this, sampling event loop lag every second. It's flat at 2ms the entire incident window. Nothing is blocking the loop.

Third theory: Postgres connection pool exhaustion, the usual suspect when latency spikes with no obvious cause. pg_stat_activity shows nine active connections against a pool of forty. The database isn't the bottleneck either.

the hunt

The pattern that breaks the case open is timing, not metrics. Every probe failure lines up with a request to /reports/export, the Excel export endpoint finance's month-end automation hits once per cost center. Their script had kicked off fifty-three export calls at concurrency 50 starting at 03:57 UTC, one minute before the first restart.

A py-spy dump on a pod mid-probe-failure shows why that matters.

Thread 0x7f2a (idle): asyncio event loop, waiting on epoll
Thread 0x7f3b (active): openpyxl.worksheet._writer, write_cell
Thread 0x7f3c (active): openpyxl.worksheet._writer, write_cell
Thread 0x7f3d (active): openpyxl.worksheet._writer, write_cell
...37 more threads in openpyxl / pandas groupby...
Thread 0x7f9e (active): anyio._backends._asyncio.WorkerThread, waiting for token
  -> run_sync(healthz)

The event loop thread is idle, just as the lag metric said. Forty other threads are busy building spreadsheets. And the liveness probe's own handler is sitting in the same queue as those forty threads, waiting for a token it never gets in time.

/reports/export was written as a plain def, not async def, because it calls openpyxl, a synchronous library, and FastAPI's own docs say this is fine: define it as a regular function and Starlette runs it in a thread pool so it never blocks the loop. /healthz was also written as a plain def, because it does nothing async and nobody thought twice about it. Both routes are correct FastAPI. Both also happen to share the same thread pool.

the find

Starlette's run_in_threadpool, the function behind every sync path operation and every sync dependency, calls anyio.to_thread.run_sync under the hood. That function pulls a token from a single CapacityLimiter, one per event loop, shared by every sync route and sync dependency running in that worker process.

>>> import anyio.to_thread
>>> limiter = anyio.to_thread.current_default_thread_limiter()
>>> limiter.total_tokens, limiter.borrowed_tokens
(40, 40)

Forty tokens, all borrowed. That ceiling isn't configured anywhere in this codebase; it's anyio's own default, unrelated to Uvicorn's worker count or any thread setting the team had tuned. Fifty-three export requests landing within seconds each held a token for four to twelve seconds while openpyxl styled cells and wrote XLSX bytes. The forty-first request to need a token, whatever route it belonged to, simply queued. On this incident, that request was /healthz, called every two seconds by the kubelet with a three-second timeout and a failure threshold of three. Three queued probes in a row and kubelet restarted a pod that was never actually unresponsive. It was waiting in the same line as a spreadsheet.

the fix

The health check never needed a thread in the first place, so the first fix removes it from the shared pool entirely.

@app.get("/healthz")
async def healthz():
    return {"status": "ok"}

Switching one keyword decouples probe health from any sync route's saturation, permanently. The second fix moves the actual blocking work, the export itself, off the request path instead of just raising the ceiling it competes under.

@app.post("/reports/export")
async def export_report(cost_center: str, db: Session = Depends(get_db)):
    job = export_queue.enqueue(build_and_store_workbook, cost_center)
    return {"job_id": job.id, "status_url": f"/reports/jobs/{job.id}"}

Finance's script now polls a job id instead of holding an HTTP connection open through the spreadsheet build, and build_and_store_workbook runs in an RQ worker process that has no liveness probe, no customer-facing routes, and nothing else competing for its threads. As a stop-gap while that queue was being wired up, the team also raised the limiter's ceiling at startup, explicitly labeled as a mitigation rather than the fix:

from anyio.to_thread import current_default_thread_limiter

@app.on_event("startup")
async def raise_thread_limiter() -> None:
    current_default_thread_limiter().total_tokens = 200

Raising the number buys headroom. It doesn't stop a large enough burst from saturating it again, which is why the export still moved to a queue regardless. The last piece is a gauge so saturation shows up before a probe ever fails:

limiter = current_default_thread_limiter()
thread_pool_saturation.set(limiter.borrowed_tokens / limiter.total_tokens)

the aftermath

Nothing in this incident was a bug in the strict sense. Every route was written the way FastAPI's own documentation recommends. The spreadsheet job finished, the health check eventually answered, and thirty-one pods got restarted anyway because two unrelated routes were quietly standing in the same line.