The FastAPI WebSocket Handler That Leaked a File Descriptor Every Time a Phone Locked

17:38 UTC. Uvicorn worker 2 starts logging OSError: [Errno 24] Too many open files on every new connection attempt. PagerDuty fires two minutes later once the error rate crosses the threshold. The launch broadcast we'd been building toward for six weeks is 38 minutes old.

Nobody on call had touched WebSocket code that week. Nothing had deployed since the previous afternoon. The only thing different about this Tuesday was that roughly 2,600 people had joined the same live channel inside a ten-minute window, which is exactly the kind of load a launch is supposed to produce and exactly the kind of load we thought we'd tested for.

the setup

The feature is a live-updates channel: users open the app, connect to a WebSocket, and get pushed real-time state during a scheduled broadcast. Four Uvicorn workers behind an internal load balancer, round-robin routing, each worker holding its own set of open connections in memory. Nothing exotic.

@app.websocket("/ws/live/{channel_id}")
async def live_channel(websocket: WebSocket, channel_id: str):
    await websocket.accept()
    connections[channel_id].add(websocket)
    try:
        while True:
            await websocket.receive_text()  # client pings aren't required by us
    except WebSocketDisconnect:
        pass
    finally:
        connections[channel_id].discard(websocket)

This had run in production for four months of smaller, quieter channels with no incident. The finally block cleans up on a normal disconnect. What it assumes, silently, is that a normal disconnect is the only kind that happens.

the scramble

First theory, ninety seconds into the incident channel: someone was hammering the connect endpoint, intentionally or not, and we were looking at a self-inflicted DDoS from launch-day traffic. We pulled connection logs for worker 2 expecting a tight cluster of retries from a small number of IPs.

We didn't find that. 2,600 distinct sessions, spread across normal mobile user agents, arriving over ten minutes at a rate that matched our own push notification send curve almost exactly. Every connection looked like a real person opening the app because we'd told them to.

Second theory: a memory leak in the broadcast fan-out, something holding references and slowly degrading the worker until it fell over. We checked RSS on worker 2 against the other three.

$ ps -o rss,vsz -p $(pgrep -f "uvicorn.*worker-2")
  RSS    VSZ
 89412 412880

$ ps -o rss,vsz -p $(pgrep -f "uvicorn.*worker-1")
  RSS    VSZ
 87904 409712

Memory on the failing worker was nearly identical to the healthy ones. Whatever was wrong wasn't consuming RAM. It was consuming something else, and the actual error message had told us what from the start. We'd just been too busy chasing traffic and memory to read it literally.

the hunt

Too many open files means file descriptors, and every open socket is one. We checked the soft limit and the actual count on the struggling worker.

$ ulimit -n
1024

$ lsof -p $(pgrep -f "uvicorn.*worker-2") | wc -l
1011

$ ss -tan state close-wait | wc -l
640

1,011 of 1,024 available file descriptors on one worker, and 640 of the open sockets sitting in CLOSE_WAIT. That state means the remote side closed its end of the TCP connection, but our process never called close() on ours. The socket is dead. The application doesn't know it yet.

That pointed straight back at the handler. await websocket.receive_text() blocks until either a message arrives or the connection raises WebSocketDisconnect. A clean disconnect, browser tab closed, app quit normally, raises that exception immediately and the finally block runs. But a mobile app that gets backgrounded doesn't always send a close frame. iOS and Android both suspend network activity on backgrounding without necessarily tearing down the socket cleanly, and several carrier NATs on our user base's networks drop the mapping silently rather than forwarding a FIN. From the server's side, nothing happens at all. receive_text() just keeps waiting, forever, for a message that will never come from a client that is already gone.

Under ordinary traffic this leaked a handful of connections a day, invisible against a ulimit of 1024. A launch pushed with a push notification is mobile-heavy by design: people tap it, glance at the update, and background the app within seconds. In the first twenty minutes, 640 of the roughly 2,600 joins on worker 2 did exactly that, and each one left a socket that our code had no way of noticing was dead.

the find

Root cause: the WebSocket handler had no idle timeout and no application-level heartbeat, so it had no way to distinguish a silent client from a slow one. It relied entirely on the transport layer to tell it when a connection had ended, and a meaningful share of mobile clients never send that signal. Every one of those connections held a file descriptor, an entry in the in-memory connections set, and an idle asyncio task, none of which would ever be released without a restart.

The 384-connection gap between worker 2's fd count and its neighbors wasn't caused by more traffic reaching that worker. Round-robin routing gave all four roughly the same join volume. It was caused by worker 2 landing, by chance, a slightly higher share of the mobile sessions that backgrounded early, and the leak compounding fast enough that the difference became visible within the same incident window instead of spreading evenly across a week like it normally would have.

the fix

The handler needed to stop assuming silence meant nothing was wrong. We added an application-level ping with a bounded wait for a pong, and treated a timeout as a disconnect worth cleaning up exactly like a real one.

PING_INTERVAL_S = 20
PONG_TIMEOUT_S = 10

@app.websocket("/ws/live/{channel_id}")
async def live_channel(websocket: WebSocket, channel_id: str):
    await websocket.accept()
    connections[channel_id].add(websocket)
    try:
        while True:
            try:
                await asyncio.wait_for(
                    websocket.receive_text(), timeout=PING_INTERVAL_S
                )
            except asyncio.TimeoutError:
                await websocket.send_text('{"type":"ping"}')
                try:
                    await asyncio.wait_for(
                        websocket.receive_text(), timeout=PONG_TIMEOUT_S
                    )
                except asyncio.TimeoutError:
                    break  # no pong: treat as dead, release the fd
    except WebSocketDisconnect:
        pass
    finally:
        connections[channel_id].discard(websocket)
        with contextlib.suppress(RuntimeError):
            await websocket.close()

Worst case, a silently dead connection now gets found and released within 30 seconds instead of sitting open indefinitely. We also raised the soft ulimit per worker from 1024 to 4096 as headroom, not as the fix, and added a gauge on open connections per worker with an alert at 70% of the limit so the next version of this problem pages someone before a worker actually falls over.

the aftermath

380 users hit connection failures during the nine minutes before the load balancer's health check started routing new joins around worker 2. Everyone already connected kept their session; it was only new arrivals during that window who saw a spinner instead of the broadcast. We restarted the worker, which cleared the leak instantly and reminded us how easy it is to mistake a restart's side effect for a real fix.

We've run two more live broadcasts since, both larger than the one that paged us. Open connection counts per worker have stayed flat and predictable through both, and the alert at 70% of ulimit hasn't fired once.