How a Login Spike Filled Node's Four-Thread Pool and Made Every Outbound Call Time Out on DNS
← Back
November 18, 2026NodeJS8 min read

How a Login Spike Filled Node's Four-Thread Pool and Made Every Outbound Call Time Out on DNS

Published November 18, 20268 min read

09:02 UTC, Monday. Datadog fires api-gateway p99 latency > 2s. Then it fires again with a bigger number. By 09:06 the p99 is 9.1 seconds and the incident channel has eleven people in it. Every pod is up, every health check is green, and the dashboard says CPU is at 38 percent.


the scramble

I get paged first because the service is mine. Support is already reporting that the mobile app spins on launch, and the payments team asks why the Stripe calls from our API are timing out when Stripe's status page is clean.

First theory: the database. Login is the heaviest thing that happens at 09:00 on a Monday, and a slow query on the users table would explain slow logins. Postgres shows 14 active connections out of 100, the slowest query in pg_stat_statements averages 3 ms, and there are no locks. Ruled out.

Second theory: the upstream calls. Outbound requests to Stripe, our email provider and the internal feature-flag service are all failing with ETIMEDOUT or hanging until our 5-second client timeout. Three unrelated vendors failing at once is not a vendor problem. I check anyway. From a debug pod in the same node group, curl to all three returns in under 150 ms.

Third theory: a bad deploy. Nothing shipped since Friday. Rolling restarts of two pods clear the problem for about 40 seconds and then it comes back, which at least tells me it is load-dependent and not a stuck state.


the hunt

curl works from a debug pod and the same host fails from the API pod. So the network is fine and something inside the Node process is not. I check the event loop first, since a blocked loop would produce exactly this symptom. The monitorEventLoopDelay histogram we export is flat: p99 loop delay is 11 ms. The loop is not blocked. It is waiting.

I attach to one pod and capture where requests spend their time. An outbound call to Stripe, traced with NODE_DEBUG=net, shows the gap before the TCP connection even starts:

09:11:42.003 NET 1: pipe/connect: dns lookup api.stripe.com requested 09:11:46.871 NET 1: lookup api.stripe.com -> 54.187.xx.xx (4868 ms) 09:11:46.874 NET 1: connect: attempting 54.187.xx.xx:443 09:11:46.921 NET 1: connected (47 ms)

The connection takes 47 ms. The DNS lookup takes 4.9 seconds. The resolver is fine, because dig from the same pod answers in 2 ms. So the lookup is slow before it ever reaches the resolver, which means it is queued.

Dead end worth mentioning: I spent ten minutes on CoreDNS, because 4.9 seconds looks like the classic 5-second resolver timeout. CoreDNS CPU is idle and its request latency is 1 ms. The number was a coincidence of our client timeout, not a resolver timeout.

What queues a DNS lookup in Node? The default dns.lookup, which every http.request uses, does not do async DNS. It calls getaddrinfo on a worker in libuv's threadpool. The pool has four threads by default. I look at what else is on that pool during a login:

// auth/login.ts import bcrypt from "bcrypt"; export async function verifyPassword(plain: string, hash: string) { return bcrypt.compare(plain, hash); // runs on the libuv threadpool }

the find

Our password hashes use bcrypt with cost 12, and a compare takes about 250 ms of pure CPU on a threadpool thread. Four threads can therefore complete 16 bcrypt compares per second per pod, and while all four are busy, nothing else that needs the pool can run. That includes getaddrinfo, fs.readFile, zlib and crypto.pbkdf2.

Monday at 09:00 is when the mobile app's saved sessions expire and the whole user base logs in again within a few minutes. The login endpoint was seeing about 90 requests per second across 6 pods, so 15 per pod. That is right at the pool's ceiling of 16, and the queue grows without bound once arrivals exceed it. Any outbound call that needed a fresh DNS lookup waited behind the bcrypt jobs. CPU at 38 percent made sense too: four busy threads on an 8-vCPU pod is half the machine, and the rest was idle.

I confirm it by measuring the pool directly with a tiny probe that schedules a trivial threadpool job and times how long it waits:

import { pbkdf2 } from "node:crypto"; setInterval(() => { const t0 = performance.now(); pbkdf2("x", "y", 1, 8, "sha1", () => { metrics.histogram("threadpool.wait_ms", performance.now() - t0); }); }, 1000);

On a healthy pod the wait is under 1 ms. During the spike it reads 4,200 to 5,100 ms, which matches the DNS gap to within noise.


the fix

Two changes: one to stop the bleeding, one to stop login hashing from sharing a pool with everything else.

First, at 09:31 we raised the pool through the pod environment and rolled the deployment. The variable has to be set before the pool is first used, so it cannot be set from application code reliably:

# deployment.yaml env: - name: UV_THREADPOOL_SIZE value: "16"

p99 dropped back to 210 ms within two minutes. It was only a mitigation. 16 threads on 8 vCPUs just moves the ceiling.

Second, bcrypt compares now go through a small dedicated worker pool, so they can no longer take threads that DNS needs:

import Piscina from "piscina"; const hashPool = new Piscina({ filename: new URL("./bcrypt-worker.js", import.meta.url).href, maxThreads: 4, maxQueue: 200, // fail fast instead of queueing forever }); export function verifyPassword(plain: string, hash: string) { return hashPool.run({ plain, hash }); }

The maxQueue matters as much as the pool. During the spike, requests waited seconds for a slot and the client retried them, which doubled the load. Now a full queue returns a 503 with a Retry-After header in a few milliseconds, and the mobile client backs off.

We also cache successful DNS results for outbound calls with cacheable-lookup, which resolves with the async resolver and does not touch the threadpool at all.


the aftermath

  • Impact: 29 minutes of elevated latency (09:02 to 09:31), with roughly 31 percent of API requests exceeding the 5-second client timeout at the peak. About 6,400 failed Stripe calls were retried successfully later by the payment reconciliation job, so no charges were lost.
  • The threadpool.wait_ms probe is now exported from every Node service, with an alert at 100 ms p95. It would have fired at 09:02.
  • The login endpoint has a load test at 150 requests per second in CI, which the old setup failed at 16 per pod.
  • Runbook line: if outbound calls time out and the event loop delay is flat, check threadpool wait before the network.

Every dashboard we had was pointed at the wrong place. The resolver was fast, the vendors were fast, and the event loop was idle. The only shared thing left was the four threads, and bcrypt was using all of them.

Share this
← All Posts8 min read