How a Login Spike Filled Node's Four-Thread Pool and Made Every Outbound Call Time Out on DNS
09:02 UTC, Monday. Datadog fires api-gateway p99 latency > 2s. Then it fires again
with a bigger number. By 09:06 the p99 is 9.1 seconds and the incident channel has eleven people
in it. Every pod is up, every health check is green, and the dashboard says CPU is at 38 percent.
the scramble
I get paged first because the service is mine. Support is already reporting that the mobile app spins on launch, and the payments team asks why the Stripe calls from our API are timing out when Stripe's status page is clean.
First theory: the database. Login is the heaviest thing that happens at 09:00 on a Monday, and a
slow query on the users table would explain slow logins. Postgres shows 14 active connections out
of 100, the slowest query in pg_stat_statements averages 3 ms, and there are no locks.
Ruled out.
Second theory: the upstream calls. Outbound requests to Stripe, our email provider and the
internal feature-flag service are all failing with ETIMEDOUT or hanging until our
5-second client timeout. Three unrelated vendors failing at once is not a vendor problem. I check anyway. From a debug pod in the same node group, curl to all three returns in
under 150 ms.
Third theory: a bad deploy. Nothing shipped since Friday. Rolling restarts of two pods clear the problem for about 40 seconds and then it comes back, which at least tells me it is load-dependent and not a stuck state.
the hunt
curl works from a debug pod and the same host fails from the API pod. So the network is fine and
something inside the Node process is not. I check the event loop first, since a blocked loop would
produce exactly this symptom. The monitorEventLoopDelay histogram we export is flat:
p99 loop delay is 11 ms. The loop is not blocked. It is waiting.
I attach to one pod and capture where requests spend their time. An outbound call to Stripe, traced
with NODE_DEBUG=net, shows the gap before the TCP connection even starts:
The connection takes 47 ms. The DNS lookup takes 4.9 seconds. The resolver is fine, because
dig from the same pod answers in 2 ms. So the lookup is slow before it ever reaches
the resolver, which means it is queued.
Dead end worth mentioning: I spent ten minutes on CoreDNS, because 4.9 seconds looks like the classic 5-second resolver timeout. CoreDNS CPU is idle and its request latency is 1 ms. The number was a coincidence of our client timeout, not a resolver timeout.
What queues a DNS lookup in Node? The default dns.lookup, which every
http.request uses, does not do async DNS. It calls getaddrinfo on a
worker in libuv's threadpool. The pool has four threads by default. I look at what else is on that
pool during a login:
the find
Our password hashes use bcrypt with cost 12, and a compare takes about 250 ms of pure CPU on a
threadpool thread. Four threads can therefore complete 16 bcrypt compares per second per pod, and
while all four are busy, nothing else that needs the pool can run. That includes
getaddrinfo, fs.readFile, zlib and crypto.pbkdf2.
Monday at 09:00 is when the mobile app's saved sessions expire and the whole user base logs in again within a few minutes. The login endpoint was seeing about 90 requests per second across 6 pods, so 15 per pod. That is right at the pool's ceiling of 16, and the queue grows without bound once arrivals exceed it. Any outbound call that needed a fresh DNS lookup waited behind the bcrypt jobs. CPU at 38 percent made sense too: four busy threads on an 8-vCPU pod is half the machine, and the rest was idle.
I confirm it by measuring the pool directly with a tiny probe that schedules a trivial threadpool job and times how long it waits:
On a healthy pod the wait is under 1 ms. During the spike it reads 4,200 to 5,100 ms, which matches the DNS gap to within noise.
the fix
Two changes: one to stop the bleeding, one to stop login hashing from sharing a pool with everything else.
First, at 09:31 we raised the pool through the pod environment and rolled the deployment. The variable has to be set before the pool is first used, so it cannot be set from application code reliably:
p99 dropped back to 210 ms within two minutes. It was only a mitigation. 16 threads on 8 vCPUs just moves the ceiling.
Second, bcrypt compares now go through a small dedicated worker pool, so they can no longer take threads that DNS needs:
The maxQueue matters as much as the pool. During the spike, requests waited seconds for a
slot and the client retried them, which doubled the load. Now a full queue returns a 503 with a
Retry-After header in a few milliseconds, and the mobile client backs off.
We also cache successful DNS results for outbound calls with cacheable-lookup, which
resolves with the async resolver and does not touch the threadpool at all.
the aftermath
- Impact: 29 minutes of elevated latency (09:02 to 09:31), with roughly 31 percent of API requests exceeding the 5-second client timeout at the peak. About 6,400 failed Stripe calls were retried successfully later by the payment reconciliation job, so no charges were lost.
- The
threadpool.wait_msprobe is now exported from every Node service, with an alert at 100 ms p95. It would have fired at 09:02. - The login endpoint has a load test at 150 requests per second in CI, which the old setup failed at 16 per pod.
- Runbook line: if outbound calls time out and the event loop delay is flat, check threadpool wait before the network.
Every dashboard we had was pointed at the wrong place. The resolver was fast, the vendors were fast, and the event loop was idle. The only shared thing left was the four threads, and bcrypt was using all of them.