How a CPU Limit That Looked Fine on Every Dashboard Throttled Search Into a 2.6-Second p99 During a Flash Sale

11:58 UTC. A flash sale goes live on the homepage. Search traffic on /api/search/rank climbs 4x in the first ninety seconds, which is what the capacity plan expected. PagerDuty fires anyway, on p99 latency, not on error rate. The ranking service isn't throwing 500s. It's just slow, 90ms baseline climbing to 2.6 seconds, and it stays there for the next twenty minutes while traffic keeps climbing.

the scramble

First theory in the incident channel: the Postgres read replica backing the product catalog lookup is the bottleneck, that's the usual suspect whenever a request path gets slow under load. Someone pulls up replica CPU and active connection count. Both flat, both nowhere near the limits that paged anyone before. Connection pool saturation isn't it either, the pool has headroom.

Second theory: the ranking model shipped two days ago is slower than the old one and nobody noticed at normal traffic. Someone checks the deploy timeline, that model has been running clean for 48 hours at steady-state load with no latency change. Someone floats rolling it back anyway. Nobody can point at anything specific wrong with the model, so it stays.

Third theory: event loop lag. The ranking service is Node, and a blocked event loop under burst traffic is a believable story. The event loop lag metric is up, but only to 40ms, nowhere close to explaining a multi-second p99. Whatever's eating the time, it isn't sitting in that number.

the hunt

Every dashboard anyone's pulled up says CPU is fine. Grafana shows each pod around 42% of its CPU limit, the HPA hasn't even scaled up because the average utilization target is 70%. That's the number everyone kept staring at and ruling CPU out because of, until someone who's been burned by this exact flavor of bug before finally asks the question nobody else had: is that utilization number averaged over a window long enough to hide something.

cAdvisor exposes a metric that never shows up on a standard CPU usage panel: how many 100ms scheduling periods got throttled by the kernel's CFS quota enforcement, versus how many ran at all.

sum(rate(container_cpu_cfs_throttled_periods_total{pod=~"search-ranking-.*"}[1m]))
/
sum(rate(container_cpu_cfs_periods_total{pod=~"search-ranking-.*"}[1m]))

# result: 0.87

87% of CPU scheduling periods for these pods were being throttled. Not 87% CPU usage, 87% of the fixed 100ms windows the kernel hands out were hitting their quota and getting frozen for the rest of the window. That's invisible to a utilization graph, because utilization is usage divided by limit averaged over the scrape interval, typically 15 to 30 seconds. A pod can burn its entire quota in the first 15ms of a 100ms period, sit frozen for the other 85ms, and still report as using a modest fraction of its limit when that's averaged out over half a minute.

The CPU limit on the deployment was 500m, set when the service first went to production and never revisited. Under the CFS quota model, a 500m limit with the kernel's default 100ms period gives each pod exactly 50ms of CPU time per 100ms window, across however many cores it tries to use. Steady-state load, one ranking request at a time, needed roughly 220 to 280m, comfortably inside that quota. The flash sale brought concurrent requests that each needed a burst of CPU to parse the candidate set and run the sort, and multiple of those bursts landing in the same 100ms window add up fast. Once the pod's combined demand crossed 50ms of work in a given period, the kernel froze it for whatever was left of that period, no matter how many idle cores sat on the node.

the find

Root cause: a CPU limit sized for average load, not burst load, hit the kernel's CFS quota wall as soon as requests started piling up concurrently. The throttling that caused was invisible to the one metric everyone was watching. The service wasn't CPU-starved in any way a percent-of-limit dashboard could show. It was CPU-starved in 100ms increments that a 15-second scrape interval smooths into a number that looks calm.

the fix

Two changes. First, the CPU limit went up to give each pod enough quota per period to absorb a burst instead of smoothing across many periods:

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2"
    memory: "512Mi"

Memory kept a tight limit, this service has no memory leak history and OOM is the one failure mode worth bounding hard. CPU limit moved from 500m to 2 full cores, giving 200ms of quota per 100ms period, enough headroom for concurrent bursts without starving the node's bin-packing entirely (removing the CPU limit outright was considered and rejected, this cluster runs mixed workloads and an unbounded burster can still starve a noisy neighbor on the same node).

Second, the throttle ratio itself got turned into an alert, since it's the only signal that would have caught this before the pager did:

- alert: CPUThrottlingHigh
  expr: |
    sum(rate(container_cpu_cfs_throttled_periods_total{namespace="search"}[5m])) by (pod)
    /
    sum(rate(container_cpu_cfs_periods_total{namespace="search"}[5m])) by (pod)
    > 0.25
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.pod }} is CPU throttled on {{ $value | humanizePercentage }} of periods"

A 25% threshold catches the problem forming well before it reaches 87%, and well before a human staring at a utilization graph would ever suspect CPU as the cause.

the aftermath

Nothing was wrong with the database, the model, or the event loop. The entire incident lived in the gap between what a CPU limit controls and what a CPU utilization dashboard can actually show you about it.