How a 30-Second Stabilization Window Scaled Us Down From 23 Pods to 17 Mid-Spike

14:31:15 UTC. PagerDuty fires for elevated 5xx rate on the checkout API. Grafana shows the error line climbing almost straight up, and the pod count panel next to it shows something that doesn't make sense at first glance: replicas dropping from 23 to 17 at the exact same second traffic started climbing, not after it.

Nine minutes later the error rate was back to baseline. Working out why the cluster removed capacity in the middle of its biggest spike of the quarter took a lot longer than fixing it did, and the actual cause traced back to a change nobody in the incident channel remembered making: a two-week-old tweak to one autoscaler setting, made for a completely different reason.

the setup

We run the checkout and catalog API on EKS, a Deployment fronted by a Horizontal Pod Autoscaler targeting 60% average CPU utilization, floor of 12 replicas, ceiling of 60. Steady state on a normal weekday afternoon sits around 18 pods. Nothing unusual about the setup, it's the default HPA v2 shape most teams end up with.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  minReplicas: 12
  maxReplicas: 60
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60

That stabilizationWindowSeconds: 30 was not the default. Two weeks earlier, on August 6th, someone on infra had shipped a cost-tuning PR lowering it from Kubernetes' own default of 300 seconds. The reasoning was sound on its own: our traffic drops off a cliff every night around 22:00 UTC, and a five-minute stabilization window meant we kept paying for peak-hour pod counts for five extra minutes after every single evening drop-off, every day. The PR description even had a dollar figure attached, a few hundred dollars a month in EC2 spend. Nobody connected that change to what a short window does during a deploy that lands in the middle of a traffic ramp, because those two events had never coincided before.

the scramble

First theory, inside the first minute: bad deploy. A routine bug fix had gone out at 14:28 UTC, three minutes before the incident started, and "we just shipped and now it's broken" is the most common shape an incident takes. Someone rolled it back immediately.

kubectl rollout undo deployment/checkout-api -n prod

The rollback completed by 14:33. The error rate kept climbing. Whatever was wrong, it wasn't the code that had just shipped, which meant the fifteen minutes everyone had spent assuming a deploy-caused bug and chasing it through the diff were fifteen minutes not spent looking at the thing that actually mattered.

Second theory: our WAF had started rate-limiting the legitimate traffic surge, mistaking a marketing email blast (sent to 380,000 subscribers at 14:30, timed almost exactly on top of the deploy by coincidence) for abuse. We pulled the WAF logs. Clean. Nothing blocked, nothing challenged. Whatever was throttling requests, it wasn't happening at the edge.

the hunt

That's when someone actually looked at the replica count graph instead of glancing past it, and said out loud what should have been obvious from the start: pod count went down while request volume was going up. That's backwards for an autoscaler that's supposed to react to load.

Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  9m    horizontal-pod-autoscaler  New size: 23; reason: cpu resource utilization above target
  Normal  SuccessfulRescale  8m    horizontal-pod-autoscaler  New size: 17; reason: All metrics below target

"All metrics below target," at 14:31:15, is the line that mattered. The HPA hadn't misfired or gotten stuck, it had done exactly what its inputs told it to do: it measured average CPU across the fleet, got a number under 60%, and cut replicas accordingly. The question became why the average was low when every dashboard showed traffic climbing.

We pulled per-pod CPU from that exact fifteen-second window. Not the fleet average, the actual distribution across individual pods.

pod-checkout-api-7f9b-h2k4x   94% cpu   (running since 13:40, pre-deploy)
pod-checkout-api-7f9b-m8q1z   91% cpu   (running since 13:40, pre-deploy)
pod-checkout-api-7f9b-p3n7w   96% cpu   (running since 13:40, pre-deploy)
pod-checkout-api-9c2d-x4r8t    9% cpu   (Ready at 14:29:52, from the deploy)
pod-checkout-api-9c2d-b7k2m   11% cpu   (Ready at 14:30:04, from the deploy)
pod-checkout-api-9c2d-w5t9q    7% cpu   (Ready at 14:30:11, from the deploy)
pod-checkout-api-9c2d-z1p6y   14% cpu   (Ready at 14:30:18, from the deploy)
pod-checkout-api-9c2d-k8m3x    8% cpu   (Ready at 14:30:26, from the deploy)

There it was, a clean bimodal split. Pods that predated the deploy were pegged near 95%, well above target on their own. Pods the rolling update had just created were reporting under 15%, not because they were idle, but because Kubernetes counted them "Ready" the instant they passed a basic liveness check, well before Service endpoint propagation had caught up and before their connection pools and JIT-warmed code paths meant they could actually carry a proportional share of load. For those first few seconds, new pods exist in the metric average as real capacity while behaving like idle capacity.

Average it all together, five near-idle pods diluting three maxed-out ones, and the fleet-wide number comes out at 42%. Below the 60% target. The HPA doesn't see individual pods, it sees one number, and that one number said "scale down."

the find

Root cause: the rolling deploy's freshly-Ready pods diluted the average CPU metric below target at the exact moment the fleet needed more capacity, and the 30-second stabilization window (down from Kubernetes' default of 300) gave the HPA no recent history to smooth that dip against. With the default five-minute window still in place, the algorithm would have kept the last five minutes of readings in view, all of which were at or above target from the traffic already building, and the single diluted sample wouldn't have been enough to trigger a rescale on its own. At 30 seconds, it was.

The scale-down from 23 to 17 pods landed at 14:31:15. Real traffic kept climbing through it. With six fewer pods than the fleet actually needed, the remaining pods saturated fast, CPU on all of them shot past 95%, and the next HPA tick 15 seconds later started scaling back up, correctly this time, but from a hole. Each new pod takes 30-45 seconds to become Ready. Clawing back from 17 to the 34 replicas the surge actually required took until roughly 14:37, five and a half minutes during which the fleet was running under-provisioned against real demand, which is where the 502s came from.

the fix

Two changes, both narrow. First, restore the stabilization window to something that can absorb a single bad sample without erasing five minutes of prior high readings. We kept the faster scale-up behavior (unaffected by this incident) and only touched scale-down.

  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60

Second, and more directly at the actual dilution mechanism: a pod shouldn't count as real capacity in the CPU average until it's had time to actually behave like real capacity. minReadySeconds on the Deployment delays a pod's contribution to rollout progress, and it also delays how long a freshly-started pod is treated as steady-state Ready for scheduling and metrics purposes.

spec:
  minReadySeconds: 45
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0

45 seconds was chosen against our own warm-up curve: pods measured in staging under synthetic load reach representative CPU behavior within about 30 seconds of receiving real traffic, so 45 gives margin without meaningfully slowing deploys. We also added a second, independent alert that doesn't depend on the HPA's own view of the world at all: page on any replica count drop greater than 20% within a two-minute window, regardless of what the utilization metric says caused it. That alert would have fired at 14:31:16, a full minute before the 502 rate crossed our existing threshold.

the aftermath

Nobody outside the infra team had known the stabilization window had changed, and the person who changed it had no reason to connect it to a rolling deploy landing mid-spike, those two things had simply never happened at the same time before this. That's the uncomfortable part of the postmortem: the change was reasonable in isolation, reviewed, and worked exactly as intended for two weeks before the one set of conditions it hadn't been tested against showed up.

We've had two more deploys land inside traffic spikes since, on purpose, as a test. Both held steady. The five-minute window costs us a few hundred dollars a month in slightly slower evening scale-down. Nine minutes of degraded checkout cost a lot more than that once, and it only has to happen once.