How a Missing Jitter on WebSocket Reconnects Turned One Dropped Task Into a 41-Minute Outage

02:47 UTC. PagerDuty: "pulse-gateway target group: 4/6 healthy." Ninety seconds later: "1/6 healthy." CPU across the whole fleet was pinned near 100%. Nobody had deployed anything since the previous afternoon.

the setup

pulse-gateway is the WebSocket service that pushes live notifications to every logged-in web and mobile client. Six ECS tasks behind an ALB, each client holds one long-lived connection, authenticated once at connect time: the gateway verifies the JWT locally, then checks a Redis-backed revocation list to make sure the session hasn't been logged out elsewhere. That Redis check runs through a single shared ioredis pool, 50 connections, sized months ago for a steady-state connect rate of roughly 20 new sessions a second.

The client SDK, a shared package used by both the web app and the mobile apps, reconnects on any unexpected close with a fixed five-second delay. It had worked fine in every prior incident, because every prior incident had dropped a handful of connections at a time, not thousands at once.

the scramble

First theory: bad deploy. The deploy log showed nothing in fourteen hours, ruled out in under a minute. Second theory: an external traffic spike, maybe a scraper or a DDoS attempt hitting the connect endpoint. ALB access logs showed the opposite of that, a wall of 101 Switching Protocols upgrade requests from client IPs that were already authenticated users, not new or unusual ones.

Third theory, and the one that actually got checked next: something was wrong with Redis itself. redis-cli --latency against the revocation-list instance showed nothing out of the ordinary, sub-millisecond round trips. Redis wasn't slow. Something in front of it was queueing.

the hunt

The connect-rate graph was the first thing that pointed anywhere useful. It didn't look like a spike, it looked like a heartbeat.

02:42  new_connections/sec  22
02:43  new_connections/sec  19
02:43  new_connections/sec  2,180   (single 200ms window)
02:48  new_connections/sec  1,940   (single 200ms window)
02:53  new_connections/sec  1,710   (single 200ms window)
02:58  new_connections/sec  1,390   (single 200ms window)

A spike every five seconds, decaying but not going away, was a synchronized retry loop, not organic traffic. The target group history explained the trigger: one task had failed its ALB health check at 02:43:04 after an unrelated memory pressure alarm on that container instance, and ALB deregistered it. That's routine. It happens most weeks and nobody pages for it. What made this one different was that the task was holding roughly 2,180 live WebSocket connections at the moment it dropped, and none of them closed gracefully, they all saw an abrupt 1006.

ws.onclose = () => {
  setTimeout(connect, 5000);
};

Every one of those 2,180 clients received its close event within roughly the same 200ms window and scheduled the exact same five-second timer. Five seconds later, all of them tried to reconnect at once. Each reconnect meant a fresh TLS handshake plus a Redis round trip to check revocation status, and 2,180 concurrent Redis requests queued hard behind a 50-connection pool sized for 20 requests a second, not two thousand in under a second.

clinic doctor against a running task ruled out a blocked event loop. What it showed instead was a huge number of pending promises, all waiting on the same starved pool. CPU climbed from handling that many concurrent TLS handshakes and JSON parses at once, not from any single slow operation. That CPU spike delayed the process's own /healthz endpoint, which lived on the same event loop behind the same queue of pending work.

app.get('/healthz', async (req, res) => {
  await redis.ping(); // shares the pool with every connect-time revocation check
  res.sendStatus(200);
});

A health check that shares a resource with the thing that's overloaded will report unhealthy right when reporting unhealthy does the most damage. ALB pulled a second task, then a third, concentrating the same retrying clients onto fewer and fewer targets while the five-second timer kept firing on schedule underneath all of it.

the find

Root cause: a fixed, unjittered reconnect delay turned one routine deregistration into a synchronized stampede, and a Redis pool sized for steady-state load, shared with the health check, became the actual bottleneck once concurrency jumped roughly a hundredfold in under a second. The health check failing under that load is what let one dropped task cascade into five.

the fix

First, the client SDK. Full jitter, not a fixed delay and not jitter added on top of a fixed delay, since either of those still clusters retries within a visible window.

let attempt = 0;

function scheduleReconnect() {
  const base = 500;
  const cap = 10_000;
  const backoff = Math.min(cap, base * 2 ** attempt);
  const delay = Math.random() * backoff; // full jitter, not fixed + random
  attempt += 1;
  setTimeout(connect, delay);
}

ws.onclose = scheduleReconnect;
ws.onopen = () => { attempt = 0; };

Second, the health check no longer depends on anything a connection stampede can starve.

app.get('/healthz', (req, res) => res.sendStatus(200));

app.get('/readyz', async (req, res) => {
  const ok = await redis.ping().then(() => true).catch(() => false);
  res.sendStatus(ok ? 200 : 503);
});

/healthz answers liveness only, no dependency, so a busy but functioning task never gets pulled from rotation for being busy. /readyz still checks Redis, but it's used for startup gating, not for deciding whether to deregister an already-running task under load.

Third, the connect path caps how much concurrent auth work it accepts rather than letting every upgrade request queue behind the same starved pool.

const acceptLimiter = new TokenBucket({ capacity: 200, refillPerSec: 100 });

server.on('upgrade', (req, socket, head) => {
  if (!acceptLimiter.tryRemove(1)) {
    socket.write('HTTP/1.1 503 Service Unavailable\r\nRetry-After: 2\r\n\r\n');
    socket.destroy();
    return;
  }
  wss.handleUpgrade(req, socket, head, onConnection);
});

A client rejected with 503 during a burst reconnects on its own jittered backoff instead of queueing invisibly on the server side. Excess load turns into a fast, explicit rejection instead of a slow, silent pileup.

the aftermath

One dropped task is not supposed to be an incident. It's supposed to be a line in a target group history that nobody looks at twice. The gap here wasn't in the load balancer or in Redis, both behaved exactly as configured. It was in a reconnect timer that had never been tested against the shape of load it would actually see the day thousands of clients lost their connection at once instead of one at a time.

The token bucket's refill rate of 100 a second isn't a permanent ceiling, it's a starting point chosen to stay comfortably under what the Redis pool can absorb. It gets revisited the next time either side of that number changes.