How Node's 5-Second keepAliveTimeout Gave Our Load Balancer a 502 on Roughly One Request in 400
16:41 UTC. The #support channel has three tickets in ten minutes, all the same sentence: "I clicked
pay and got a blank error page, then it worked the second time." Datadog shows the public ALB
returning 502 on 0.25% of requests to orders-api. The error budget alert fires at
16:48. Every pod in the target group reports zero 5xx.
the scramble
Two engineers get paged and the first theory arrives within a minute: a bad deploy. There was one at 14:10, a dependency bump. Rolling it back takes six minutes and the 502s continue at the same rate.
Second theory: pods are being killed mid-request. That would be consistent with a low-rate 502, and we have been bitten by it before. The restart counts say otherwise. All eight pods have been up for nine days with zero restarts, so nothing is terminating, and nothing is being deregistered either.
Third theory: the app is crashing on a specific request and returning garbage. This one dies quickly too. If the app crashed, the pod's own 5xx counter and its error logs would show it. Both are empty. The ALB says it returned 502. The pods say they never failed a request. Someone in the channel writes "one of them is lying," and nobody can say which.
the hunt
The ALB access logs are the only place both sides of the story meet. Pulling a 10-minute window out of S3 and filtering on the 502s:
The interesting columns are elb_status_code (502) and target_status_code.
On every one of these lines target_status_code is -. The target never
returned a status. That is not an application error, that is the connection to the target failing
before any response existed. CloudWatch agrees: HTTPCode_ELB_502_Count is climbing
while HTTPCode_Target_5XX_Count is flat at zero.
A failed connection should be correlated with something, so the next check is time. If the 502s cluster at a time of day, it is a cron or a batch job. They don't. If they cluster on one pod, it is one bad node. They don't, they spread across all eight roughly evenly. The only variable that does line up is traffic volume: during the overnight lull the 502 rate rises to about 1 in 150 requests, and at peak it falls below 1 in 800. Fewer requests, more failures, which is backwards for almost every overload story.
Backwards is a useful clue. Low traffic means connections sit idle between requests. The ALB keeps connections to targets open and reuses them, so an idle connection is the one most likely to get closed by the other side right as it is reused. Next step: a packet capture on one pod during the quiet period:
Twenty minutes of capture, then a look at the sequences where a request packet from the ALB's
address arrived right behind a FIN from the pod. There are 31 of them. In each case the pod sent
FIN, and the ALB's data segment, carrying a full POST /orders, arrived within about a millisecond
and was answered with RST. The pod had closed the connection and the ALB hadn't processed that yet.
the find
Node's HTTP server closes idle keep-alive connections after server.keepAliveTimeout,
which defaults to 5000 ms. The ALB's idle timeout is 60 seconds by default. So for any connection that goes
quiet, Node sends FIN at the 5 second mark and the ALB, which believes the connection is good for
55 more seconds, keeps it in its pool.
Most of the time that is harmless. The FIN arrives, the ALB notices, and the connection is
discarded. The failure is a race. If the ALB picks that pooled connection for a new request at
almost exactly the 5 second mark, its request leaves before the FIN has been processed on its side. Node has already decided the
socket is closed, so the request is answered with RST. The ALB has no safe way to retry a
non-idempotent request, so it returns 502 to the client and logs target_status_code as -.
This also explains the backwards traffic pattern. At peak, connections are reused every few milliseconds and never get near the 5 second mark. At 3 AM, a connection is idle for several seconds between requests, which puts the next request squarely in the danger window. The window itself is narrow, but at low volume nearly every reuse lands in it.
I checked that the default is what we were actually running, since the service wraps Express and I didn't trust my memory of what the wrapper set:
the fix
The rule is that the server behind a load balancer must keep idle connections open longer than the
load balancer does, so the load balancer is always the side that closes first. Our ALB idle timeout is 60
seconds, so the target needs to be higher. I set 65 seconds. headersTimeout has to be larger than
keepAliveTimeout, or Node can still cut a connection while it waits for the next
request's headers, so that goes to 66:
I put the ALB's value in a shared constant with a comment pointing at the Terraform
idle_timeout attribute, because the two numbers are only correct as a pair and
nothing enforces that. It shipped at 18:20. The 502 rate went from 0.25% to zero across the next
four hours, including the overnight lull, which is when it had been worst.
the aftermath
- Customer impact: roughly 1,900 failed requests over about 26 hours before the fix, 310 of them payment submissions. Customers who retried succeeded, so nothing was double-charged, but 41 abandoned the checkout.
- Root cause category: two layers with independent idle timeouts, ordered the wrong way round, with the default on the Node side silently winning.
- Added a CloudWatch alarm on
HTTPCode_ELB_502_Countdivided by total requests, separate from the target 5xx alarm. Our only 5xx alert watched target errors, so it would never have fired for this. - Audited the other four Node services behind that ALB. Three had the same defaults. All were fixed in the same change.
- Added a startup assertion in our shared server bootstrap that fails if
keepAliveTimeoutis below a configured load balancer idle timeout. - Runbook entry: a 502 with
target_status_codeof-and clean pod logs means the connection failed, not the app. Check idle timeouts before checking code.
We burned the first hour on three theories that all assumed the app had done something wrong, because that is where we usually look. The pods were telling the truth. They had never returned a bad response. They had just hung up on a connection a few seconds before the load balancer expected them to.