How a 30-Second Pod Grace Period Lost the Race Against a 300-Second ALB Deregistration Delay

14:32 UTC. PagerDuty fires on the checkout service's error-rate alarm. Datadog shows the 502 rate on POST /checkout jumping from a baseline of 0.02% to 4.8%, right in the middle of a rolling deploy that had shipped a dozen times before without anyone noticing it. By 14:36 the deploy finishes, the 502s stop, and the dashboard looks normal again. Nobody asked for this one to rotate twelve pods for a routine dependency bump, and yet here it was, the third deploy that week to show the exact same spike, each time clean before and after, dirty for about four minutes in the middle.

the scramble

First theory in #incidents: the new image is bad. That's the obvious read when a deploy and an error spike land on top of each other. Someone pulls up the new revision's pod logs looking for stack traces, crash loops, anything. Nothing. Zero restarts across all twelve pods. Zero 5xx logged by the app itself. Whatever's producing these 502s, it isn't the checkout service throwing an error, because the checkout service never saw the request at all.

Second theory: a health check regression is marking targets unhealthy mid-rollout and the ALB is handling the transition badly. Someone checks the target group in the AWS console. Zero unhealthy targets logged for the entire deploy window. By every signal AWS itself can show, nothing was ever wrong with any target. That's the part that stalls the room for a few minutes, because if the load balancer never thought a target was unhealthy, where are the 502s even coming from?

the hunt

With both easy theories dead, someone pulls the ALB's own access logs from S3 and greps for the deploy window instead of trusting the target-group health view. ALB access logs record a target_status_code of - for exactly one case: it picked a target, tried to open the connection, and got nothing back, connection refused or reset. That's a different failure mode than "target failed its health check," and it doesn't show up on the target group's health dashboard at all.

aws s3 cp s3://alb-logs/checkout/2026/10/20/ . --recursive
zcat *.log.gz | awk '$8 == "-" && $6 ~ /checkout/' | cut -d' ' -f4,8,14

14:33:02.418481Z - 10.4.21.117:3000
14:33:04.901223Z - 10.4.21.117:3000
14:33:05.550112Z - 10.4.21.117:3000
14:34:11.009887Z - 10.4.21.94:3000
14:34:12.733410Z - 10.4.21.94:3000

Every failed request maps to a target IP, and those IPs belong to pods. Cross-reference that against kubectl get events history for the same window: each burst of failures lines up within a handful of seconds of that exact pod's termination. 10.4.21.117 got its SIGTERM at 14:33:00. The last 502 mapped to it is at 14:33:05.55, five seconds after the process was told to shut down.

That's the actual question now: why was the ALB still sending live traffic to a pod five seconds after Kubernetes told it to die? Checking the AWS Load Balancer Controller's own logs for the same window answers it. The controller doesn't poll for pod state, it watches EndpointSlice updates and reacts by calling DeregisterTargets on the target group. The controller's reconcile loop timestamps show a four-to-seven-second gap between the pod going NotReady and the target group actually being told to drop that IP. The checkout app had no SIGTERM handler, so Node's default behavior kicked in: no listener means the process exits almost immediately, well under a second, as soon as the signal arrives.

The pod was dying in under a second. The signal that it was dying took four to seven seconds to reach the one system still routing traffic to it. The 300-second deregistration_delay.timeout_seconds on the target group, set months earlier to drain long-lived connections gracefully once a target is actually marked for removal, was never the problem and never fired, because the controller hadn't called DeregisterTargets yet when the pod went away. That setting governs what happens after deregistration starts. It has nothing to say about the gap before it starts.

the find

Root cause: the checkout app shut down faster than Kubernetes' own control plane could tell the load balancer it was going away. A bare SIGTERM with no handler killed the process in under a second. In this cluster it took four to seven seconds for that pod's IP to drop from the EndpointSlice, for the AWS Load Balancer Controller to notice, and for DeregisterTargets to actually fire on the ALB. Every request the ALB routed to that pod during that gap hit a socket nobody was listening on anymore.

the fix

Two changes, working together. First, the app needed to survive SIGTERM long enough to stop accepting new connections on its own terms instead of just dying:

const server = app.listen(3000);

process.on('SIGTERM', () => {
  console.log('SIGTERM received, draining connections');
  server.close(() => {
    console.log('server closed, no in-flight requests remain');
    process.exit(0);
  });

  // belt and suspenders: force exit if close() hangs on a stuck connection
  setTimeout(() => process.exit(1), 25_000).unref();
});

Second, and this is the half that actually fixes the race: the pod needed to delay sending SIGTERM in the first place, long enough for the EndpointSlice update and the ALB controller's deregister call to actually land before the process started shutting down.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: checkout
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 10"]
      readinessProbe:
        httpGet:
          path: /healthz
          port: 3000
        periodSeconds: 5
        failureThreshold: 1

The preStop hook runs before SIGTERM is ever sent to the container, and the pod is marked NotReady the moment termination begins, well before the hook finishes sleeping. Ten seconds was comfortably above the four-to-seven-second gap measured in the controller logs. terminationGracePeriodSeconds went up to 45 so the sleep plus the app's own graceful shutdown both have room to finish before Kubernetes gives up and sends SIGKILL.

the aftermath

Nothing about the checkout service's code was ever broken, and nothing about the ALB's configuration was wrong on its own terms. The bug lived entirely in the few seconds between two correct systems finding out about each other.