October 17, 2026Architecture9 min read

How a Fraud-Check Call Without Keep-Alive Exhausted a NAT Gateway's 55,000 Ports During a Flash Sale

Published October 17, 20269 min read

11:58 UTC, Friday. Checkout error rate crosses 1% and keeps climbing, right as the flash sale banner goes live on the homepage. The on-call engineer's phone lights up with PagerDuty before the Grafana tab even finishes loading. The errors are not a clean 500. They're timeouts, scattered across otherwise healthy requests, and they're getting worse by the minute.

the scramble

The checkout flow calls a fraud-scoring Lambda before authorizing any payment, one HTTPS call per order to a third-party fraud API, plus a quick velocity check against an RDS read replica. It lives in a private subnet, no public IP, egress routed through a single NAT gateway. First theory in the incident channel: the fraud vendor is down or rate-limiting the account. Their status page is green. A manual curl from a bastion host to the same endpoint returns a normal response in 80ms, and the account's own rate-limit headers from a successful call show plenty of headroom left.

Second theory, and the one everyone expects given the system's history: RDS connection exhaustion. This exact function got burned by that eighteen months ago, which is why it already sits behind RDS Proxy. Checking the proxy's connection metrics rules it out fast, active connections are at 60% of the configured max, nowhere near saturated. Whatever's timing out, it isn't the database call.

Both obvious suspects are dead within ten minutes, and the error rate is past 8% and still climbing. Not the vendor. Not the database. Something between the Lambda and the outside world is the actual bottleneck, and nobody on the call has a dashboard pointed at it yet.

the hunt

CloudWatch Logs for the function show the pattern clearly once someone filters for the actual error instead of the generic 5xx: connect ETIMEDOUT, thrown from the HTTPS client trying to reach the fraud API, not from anything RDS-related. That's new information. A database timeout and a TCP connect timeout to an external host are different failures with different causes, and the team had been debugging the wrong one for ten minutes because "checkout is timing out" sounds like a database story by default.

The function sits in a VPC, which means every outbound call to the public internet, including the fraud API, goes through the NAT gateway attached to its route table. NAT gateways aren't infinite. Each one allows up to 55,000 simultaneous connections per unique combination of destination IP, destination port, and protocol. Every single fraud-check call in this system goes to the same vendor endpoint, same IP, same port 443, same protocol, which means all of them share one 55,000-connection budget no matter how many different Lambda execution environments are making the calls.

the metric nobody had an alarm on

aws cloudwatch get-metric-statistics \
  --namespace AWS/NATGateway \
  --metric-name ErrorPortAllocation \
  --dimensions Name=NatGatewayId,Value=nat-0f2a9c1b8e4d7f3a1 \
  --start-time 2026-10-16T11:30:00Z --end-time 2026-10-16T12:30:00Z \
  --period 60 --statistics Sum

# Sum climbs from 0 to several thousand per minute, in lockstep
# with the checkout error graph, starting at 11:54 UTC

ErrorPortAllocation counts the number of times the NAT gateway had a connection request and no port left to give it for that destination. It had been sitting at zero for months, which is exactly why nobody had an alarm on it. The flash sale pushed this function from a baseline of roughly 40 concurrent executions to over 1,800 within three minutes of the banner going live, and every one of those invocations was making a fresh connection attempt to the same destination IP at the same time.

the find

The actual bug was in how the fraud-check client got constructed. It was being created inside the handler, on every invocation, instead of once per execution environment:

fraud-check handler, as shipped

exports.handler = async (event) => {
  const agent = new https.Agent({ keepAlive: false });
  const result = await checkFraud(event.order, { agent });
  return result;
};

keepAlive: false tells Node to close the TCP connection the moment the response finishes. Creating the agent inside the handler means even a warm Lambda execution environment, one that's already running and could reuse a socket from the last invocation, throws its connection away and opens a new one every single time. A closed connection doesn't free its NAT gateway port immediately either, it sits in TIME_WAIT for a couple of minutes per the OS's TCP stack, still counted against the destination's port budget. Under the sale's normal mid-morning traffic this cost a few extra milliseconds of TLS handshake per request and nobody noticed. Under 1,800 concurrent executions, each opening and immediately discarding a connection to one IP, the TIME_WAIT backlog caught up with the 55,000-port ceiling in under four minutes.

the fix

The client needed to live outside the handler, at module scope, so the same keep-alive connection survives across warm invocations instead of getting rebuilt every time:

fraud-check handler, after

const agent = new https.Agent({ keepAlive: true, maxSockets: 10 });

exports.handler = async (event) => {
  const result = await checkFraud(event.order, { agent });
  return result;
};

That single change cut new-connection volume to the fraud API by roughly 90% once redeployed, because most invocations inside a given execution environment now reuse the same socket instead of negotiating a new TLS handshake. It doesn't remove the ceiling, though, it just raises how much concurrency the existing NAT gateway can absorb before hitting it again. The team added a second NAT gateway in a different AZ and split the function's subnets across both route tables, so the same destination's 55,000-port budget is now doubled across two gateways instead of shared by one. A CloudWatch alarm on ErrorPortAllocation above zero, sustained for two data points, now pages on-call before checkout's error rate does.

the aftermath

14 min From first PagerDuty alert to the port-allocation metric confirming root cause

11% Peak checkout error rate during the four worst minutes of the sale

1,800+ Concurrent executions of the fraud-check function at the error spike's peak

2 NAT gateways now splitting the same destination's port budget, up from 1

A connect timeout to an external host and a database timeout look identical from the outside, "checkout is slow," but point at completely different layers. Reading the specific error string before picking a theory would have saved the first ten minutes.
A NAT gateway's 55,000-connection limit is scoped per destination IP, port, and protocol, not per gateway overall. A system that makes most of its outbound calls to one external endpoint has a much lower effective ceiling than the headline number suggests.
keepAlive: false, or any client built fresh inside a Lambda handler, throws away the one advantage a warm execution environment gives you for free. The fix costs one line and a change in scope, not a new service.
If a resource has never once triggered its own CloudWatch metric, that's not proof it's safe, it's proof nobody has watched it at the traffic level that would prove otherwise. ErrorPortAllocation existed the entire time this account had NAT gateways.

The fraud API was never the problem. The database was never the problem. The actual bottleneck was a resource nobody on the team had looked at in months, sized correctly for every day the sale wasn't running, and it took a single uninitialized keep-alive flag to find its limit in front of the busiest four minutes of the quarter.