How Node's 5-Second keepAliveTimeout Gave Our Load Balancer a 502 on Roughly One Request in 400
← Back
November 16, 2026NodeJS8 min read

How Node's 5-Second keepAliveTimeout Gave Our Load Balancer a 502 on Roughly One Request in 400

Published November 16, 20268 min read

16:41 UTC. The #support channel has three tickets in ten minutes, all the same sentence: "I clicked pay and got a blank error page, then it worked the second time." Datadog shows the public ALB returning 502 on 0.25% of requests to orders-api. The error budget alert fires at 16:48. Every pod in the target group reports zero 5xx.


the scramble

Two engineers get paged and the first theory arrives within a minute: a bad deploy. There was one at 14:10, a dependency bump. Rolling it back takes six minutes and the 502s continue at the same rate.

Second theory: pods are being killed mid-request. That would be consistent with a low-rate 502, and we have been bitten by it before. The restart counts say otherwise. All eight pods have been up for nine days with zero restarts, so nothing is terminating, and nothing is being deregistered either.

Third theory: the app is crashing on a specific request and returning garbage. This one dies quickly too. If the app crashed, the pod's own 5xx counter and its error logs would show it. Both are empty. The ALB says it returned 502. The pods say they never failed a request. Someone in the channel writes "one of them is lying," and nobody can say which.


the hunt

The ALB access logs are the only place both sides of the story meet. Pulling a 10-minute window out of S3 and filtering on the 502s:

aws s3 cp s3://acme-alb-logs/AWSLogs/.../16.40Z_xxxx.log.gz - | gunzip \ | awk '$9 == 502 {print $1, $2, $5, $9, $10, $11, $12}' | head

The interesting columns are elb_status_code (502) and target_status_code. On every one of these lines target_status_code is -. The target never returned a status. That is not an application error, that is the connection to the target failing before any response existed. CloudWatch agrees: HTTPCode_ELB_502_Count is climbing while HTTPCode_Target_5XX_Count is flat at zero.

A failed connection should be correlated with something, so the next check is time. If the 502s cluster at a time of day, it is a cron or a batch job. They don't. If they cluster on one pod, it is one bad node. They don't, they spread across all eight roughly evenly. The only variable that does line up is traffic volume: during the overnight lull the 502 rate rises to about 1 in 150 requests, and at peak it falls below 1 in 800. Fewer requests, more failures, which is backwards for almost every overload story.

Backwards is a useful clue. Low traffic means connections sit idle between requests. The ALB keeps connections to targets open and reuses them, so an idle connection is the one most likely to get closed by the other side right as it is reused. Next step: a packet capture on one pod during the quiet period:

sudo tcpdump -i eth0 -nn 'tcp port 8080 and (tcp[tcpflags] & (tcp-fin|tcp-rst) != 0)' -w /tmp/fin.pcap

Twenty minutes of capture, then a look at the sequences where a request packet from the ALB's address arrived right behind a FIN from the pod. There are 31 of them. In each case the pod sent FIN, and the ALB's data segment, carrying a full POST /orders, arrived within about a millisecond and was answered with RST. The pod had closed the connection and the ALB hadn't processed that yet.


the find

Node's HTTP server closes idle keep-alive connections after server.keepAliveTimeout, which defaults to 5000 ms. The ALB's idle timeout is 60 seconds by default. So for any connection that goes quiet, Node sends FIN at the 5 second mark and the ALB, which believes the connection is good for 55 more seconds, keeps it in its pool.

Most of the time that is harmless. The FIN arrives, the ALB notices, and the connection is discarded. The failure is a race. If the ALB picks that pooled connection for a new request at almost exactly the 5 second mark, its request leaves before the FIN has been processed on its side. Node has already decided the socket is closed, so the request is answered with RST. The ALB has no safe way to retry a non-idempotent request, so it returns 502 to the client and logs target_status_code as -.

This also explains the backwards traffic pattern. At peak, connections are reused every few milliseconds and never get near the 5 second mark. At 3 AM, a connection is idle for several seconds between requests, which puts the next request squarely in the danger window. The window itself is narrow, but at low volume nearly every reuse lands in it.

I checked that the default is what we were actually running, since the service wraps Express and I didn't trust my memory of what the wrapper set:

const server = app.listen(8080); console.log(server.keepAliveTimeout, server.headersTimeout); // 5000 60000

the fix

The rule is that the server behind a load balancer must keep idle connections open longer than the load balancer does, so the load balancer is always the side that closes first. Our ALB idle timeout is 60 seconds, so the target needs to be higher. I set 65 seconds. headersTimeout has to be larger than keepAliveTimeout, or Node can still cut a connection while it waits for the next request's headers, so that goes to 66:

const server = app.listen(8080); // ALB idle timeout is 60s. Node must outlast it, never the reverse. server.keepAliveTimeout = 65_000; server.headersTimeout = 66_000;

I put the ALB's value in a shared constant with a comment pointing at the Terraform idle_timeout attribute, because the two numbers are only correct as a pair and nothing enforces that. It shipped at 18:20. The 502 rate went from 0.25% to zero across the next four hours, including the overnight lull, which is when it had been worst.


the aftermath

  • Customer impact: roughly 1,900 failed requests over about 26 hours before the fix, 310 of them payment submissions. Customers who retried succeeded, so nothing was double-charged, but 41 abandoned the checkout.
  • Root cause category: two layers with independent idle timeouts, ordered the wrong way round, with the default on the Node side silently winning.
  • Added a CloudWatch alarm on HTTPCode_ELB_502_Count divided by total requests, separate from the target 5xx alarm. Our only 5xx alert watched target errors, so it would never have fired for this.
  • Audited the other four Node services behind that ALB. Three had the same defaults. All were fixed in the same change.
  • Added a startup assertion in our shared server bootstrap that fails if keepAliveTimeout is below a configured load balancer idle timeout.
  • Runbook entry: a 502 with target_status_code of - and clean pod logs means the connection failed, not the app. Check idle timeouts before checking code.

We burned the first hour on three theories that all assumed the app had done something wrong, because that is where we usually look. The pods were telling the truth. They had never returned a bad response. They had just hung up on a connection a few seconds before the load balancer expected them to.

Share this
← All Posts8 min read