How One Email Regex Froze a Node.js Event Loop and Took Down Every Endpoint on the Pod
← Back
November 13, 2026NodeJS8 min read

How One Email Regex Froze a Node.js Event Loop and Took Down Every Endpoint on the Pod

Published November 13, 20268 min read

09:14 UTC. PagerDuty: accounts-api liveness probe failing on 3 of 6 pods. Traffic is flat. Memory is flat. CPU on the failing pods is pinned at one full core and has been for four minutes. Kubernetes is already killing them, and the replacements are coming up healthy and then dying again.


the scramble

Nobody has deployed since Tuesday. That rules out the usual first theory, so the channel moves to the second one: a noisy neighbor. The nodes are shared with the batch workers, and a report job had misbehaved before. On-call cordons the node hosting two of the failing pods. The pods move, and they fail on the new node too.

Third theory: a dependency is slow and requests are piling up. The Postgres dashboard is quiet, Redis is quiet, and the upstream auth provider reports green. If requests were piling up on a slow dependency, memory would climb with the in-flight queue. It doesn't. The pods are busy, but not with anything that looks like waiting.

What stands out is the shape of the failure. A pod is fine, then it is at 100% CPU, then it stops answering /healthz entirely. Not slow, not erroring. Silent. A health endpoint that returns a constant string cannot be slow unless the thread that serves it is busy.


the hunt

Node runs JavaScript on one thread, so a pod that cannot answer a trivial route is a pod whose event loop is stuck inside something synchronous. The question is what. On-call wants a CPU profile from a failing pod before the probe kills it:

# pod started with NODE_OPTIONS="--cpu-prof --cpu-prof-dir=/tmp" # profile is only written when the process exits cleanly kubectl exec accounts-api-7d9f-xk2lp -- kill -SIGINT 1 kubectl cp accounts-api-7d9f-xk2lp:/tmp ./profiles

The first attempt returns nothing. The probe killed the process before it flushed a profile, and a SIGKILLed Node process writes nothing. That dead end costs eleven minutes. The second attempt patches the liveness probe out of one pod so it can stay wedged long enough to profile.

The profile has one frame that owns 99.8% of the time: RegExp.test, called from validateEmail in lib/validators.js, called from the POST /v1/signup handler. Not a database call or JSON parsing. A regular expression.

// lib/validators.js, unchanged for 22 months const EMAIL_RE = /^([a-z0-9]+[._-]?)+@[a-z0-9.-]+\.[a-z]{2,}$/i; export function validateEmail(value) { return EMAIL_RE.test(value); }

The suspicious part is ([a-z0-9]+[._-]?)+. A group containing a +, wrapped in another +, where the inner separator is optional. A run of letters can be split between iterations of the group in exponentially many ways. When the match succeeds, the engine takes the first split and never notices. When it fails, it tries all of them.

Reproducing it takes thirty seconds on a laptop with Node 22. The input is letters followed by one character that cannot match the rest of the pattern:

const re = /^([a-z0-9]+[._-]?)+@[a-z0-9.-]+\.[a-z]{2,}$/i; for (const n of [20, 24, 26, 28, 30]) { const s = "a".repeat(n) + "!"; const t = process.hrtime.bigint(); re.test(s); console.log(n, Number(process.hrtime.bigint() - t) / 1e6, "ms"); } // 20 8.5 ms // 24 22.4 ms // 26 88.2 ms // 28 357.0 ms // 30 1427.6 ms

Every two extra characters multiplies the time by four. Extrapolating that curve: 36 characters is roughly 90 seconds, and 40 characters is on the order of 20 minutes. The signup form had no length limit on the email field, and the API had none either.

why it only hit some pods

The access log for the failing pods shows the same source IP hitting /v1/signup with a 44-character value in email. It was a user whose password manager had pasted the wrong clipboard entry, followed by a client retry policy that resubmitted on timeout. Each retry landed on a different pod, froze it, and got the pod killed. The liveness probe was doing its job. The restarts were not the problem, they were the symptom of a request that kept coming back.


the find

This is catastrophic backtracking. V8's regex engine backtracks, so a pattern with nested unbounded quantifiers has exponential worst-case time on inputs that almost match. Nothing in our load tests would have caught it, because load tests send valid emails. A valid email matches on the first path and returns in microseconds.

It also explains why this was a full outage and not one slow request. JavaScript is single-threaded. One regex call running for several minutes is several minutes in which that pod accepts no connections, answers no probes, and runs no timers. A timeout on the HTTP handler does nothing, because the timer callback cannot fire while the loop is busy.


the fix

Two changes. First, a length cap before any regex runs, because RFC 5321 limits an address to 254 characters and nobody legitimate is near it. Second, a rewrite of the pattern so that each character can be matched by exactly one part of it. The separator is now required to be followed by an alphanumeric, which removes the ambiguity:

const EMAIL_RE = /^[a-z0-9]+(?:[._-][a-z0-9]+)*@[a-z0-9-]+(?:\.[a-z0-9-]+)*\.[a-z]{2,}$/i; const MAX_EMAIL_LENGTH = 254; export function validateEmail(value) { if (typeof value !== "string" || value.length > MAX_EMAIL_LENGTH) { return false; } return EMAIL_RE.test(value); }

Against the same kind of hostile input, the new pattern on a 5,000-character string of letters and a trailing ! finishes in 0.16 ms. An alternating a.a.a. string of the same size takes 0.04 ms. The time is linear in the input and the length cap bounds it anyway.

The deploy went out at 10:02. The pods that had been restarting 6 times an hour stopped. The offending client was still retrying, and now got a 400 in under a millisecond.


the aftermath

  • Customer impact: 48 minutes of intermittent failures on every route served by affected pods, since a wedged pod drops all traffic and not just signup. Roughly 3,100 failed requests, of which 2,900 were unrelated to signup.
  • Root cause category: user-controlled input reaching an unbounded-time operation on a shared single thread.
  • Added safe-regex2 to CI, run over every regex literal in the repo. It flagged 4 more patterns, 2 of them in code that handles user input. Both were rewritten the same week.
  • Added an event-loop lag metric (perf_hooks.monitorEventLoopDelay) with an alert at 200 ms p99. We would have seen this 14 minutes earlier with it.
  • Added request body field length limits at the gateway, so oversized fields are rejected before a Node process sees them.
  • Runbook entry: if a pod is at 100% CPU with a flat request rate and fails probes, take a CPU profile on a pod with the probe patched out before doing anything else. We lost eleven minutes learning that the probe kills the evidence.

The lesson I keep is narrow. I'd read that regex a dozen times and never flagged it. It matched every email we tried, so review passed it, and it ran for 22 months. A pattern can be right on every input you test and still have a worst case that waits for someone to paste the wrong thing.

Share this
← All Posts8 min read