How a Debug Label on One Histogram Grew Prometheus to 3.9 Million Series and Silenced Our Pager

11:14 UTC. A message in #support-escalations, not PagerDuty: "checkout is hanging for a full minute for at least six customers, is something down?" On-call checked their phone. No pages. No missed calls. Grafana's checkout dashboard was open in a browser tab from an hour earlier, every panel on it now said "No data."

the setup

checkout-api emits a histogram, checkout_latency_seconds, scraped by a single Prometheus replica running in the same Kubernetes cluster. No Thanos, no Cortex, one StatefulSet with a 6Gi memory limit that had been enough for two years of steady growth. Alertmanager ran as a sidecar reading rules evaluated by that same Prometheus instance. It was the one thing standing between a broken checkout flow and a page, and it was also a single point of failure nobody had written down as one.

Five hours earlier, a support ticket had come in about one customer seeing consistently slow checkouts. An engineer added a label to help narrow it down.

checkout_latency = Histogram(
    "checkout_latency_seconds",
    "Checkout request latency",
    ["customer_id", "payment_method"],  # customer_id added to debug ticket SUP-4471
    buckets=[0.1, 0.25, 0.5, 1, 2.5, 5, 10],
)

It was meant to ship for an hour, get a graph filtered to one customer, then get reverted. It didn't get reverted. It went out with the next deploy to every pod, which meant every customer who hit checkout started contributing their own label value, multiplied by seven latency buckets and two payment methods, to a metric that used to have a few hundred series total.

the scramble

First theory: Grafana itself was broken, a bad panel query or a stuck browser tab. A fresh incognito load of the dashboard showed the same blanks. Grafana's own health endpoint returned 200. The dashboard wasn't broken. It had nothing to show.

Second theory: a network problem between Grafana and its Prometheus datasource. On-call exec'd into the Grafana pod and curled Prometheus directly.

$ curl -sS http://prometheus.monitoring.svc:9090/-/healthy
curl: (7) Failed to connect to prometheus.monitoring.svc port 9090: Connection refused

Not a network problem. Prometheus wasn't listening on that port at all, which meant it probably wasn't running.

the hunt

kubectl get pods -n monitoring showed the answer nobody wanted: prometheus-0 0/1 CrashLoopBackOff 14 (78s ago). Fourteen restarts, roughly ninety seconds apart, which matched almost exactly how long the pod had been unreachable, and probably how long the pager had been silent.

level=info msg="Starting WAL replay" ...
level=info msg="WAL segment loaded" segment=00001862 ...
level=info msg="WAL segment loaded" segment=00001863 ...
(no further output, container terminated)

Last State: Terminated
  Reason: OOMKilled
  Exit Code: 137

Dead end one: WAL corruption from a previous unclean shutdown. The team ran promtool tsdb analyze against a copy of the volume. The WAL parsed fine. Whatever was killing the process was happening during normal replay, not because the data was broken.

Dead end two: someone had quietly lowered the memory limit in a recent Helm values change. git log -p on the monitoring chart showed the limit unchanged for four months at 6Gi. The ceiling hadn't moved. Something underneath it had grown.

One thing survived the crash: checkout-api also remote-writes a metrics subset to a managed Prometheus workspace for long-term retention, and that path doesn't need the local replica to stay alive, only to have already shipped its last few samples. Querying prometheus_tsdb_head_series against the managed workspace, right up to the last successful remote-write before the crash loop began, told the real story.

06:00  14,220
07:00  61,880
08:00  340,510
09:00  1,116,900
10:00  2,703,440
10:47  3,912,600   (last sample before remote-write itself stopped)

A metric that starts at fourteen thousand series and passes three point nine million in under five hours is not organic traffic growth. The timestamp lined up with checkout-api's deploy history, right at the 06:20 UTC release, and a two-line diff in its instrumentation file confirmed it: the customer_id label, still live, still collecting a brand new series for every distinct customer who checked out for the first time that morning.

the find

Root cause: an unbounded label on an existing histogram turned a low-cardinality metric into one with a new series for every customer, multiplied across every bucket and payment method combination. Each series costs Prometheus a few kilobytes of resident memory in the head block, so going from roughly three hundred series to nearly four million pushed the process well past its 6Gi limit. Kubernetes OOM-killed it, and every restart had to replay a WAL that kept the full history of that same exploding series count, so replay itself started consuming enough memory to trigger another OOM before it finished. There was no way out of the loop without intervention, the crash was recreating the exact condition that caused it every time.

The part that turned a metrics problem into a full incident: Alertmanager evaluated its rules through that same dead Prometheus instance, so no alert fired for Prometheus being down, and no alert fired for the real checkout latency spike that started independently around 10:50 UTC. Both were invisible from inside the same monitoring stack that was supposed to catch them.

the fix

Immediate mitigation, before any app code could redeploy: drop the offending label at scrape time so Prometheus could start and stay up.

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'checkout_latency_seconds.*'
    action: labeldrop
    regex: 'customer_id'

Then the actual application fix, removing the label at the source instead of relying on relabeling to keep hiding it.

checkout_latency = Histogram(
    "checkout_latency_seconds",
    "Checkout request latency",
    ["payment_method"],
    buckets=[0.1, 0.25, 0.5, 1, 2.5, 5, 10],
)

# Per-customer debugging goes through structured logs and trace attributes,
# never through a metric label. Traces are cardinality-safe by design.

A per-target sample limit went in as a blast-radius control, so the next unbounded label fails loudly on one scrape instead of taking down the whole process.

scrape_configs:
  - job_name: checkout-api
    sample_limit: 20000
    static_configs:
      - targets: ["checkout-api:9100"]

And a dead man's switch that doesn't route through Prometheus at all: a cron job pings an external heartbeat endpoint every minute, and a missed heartbeat pages on-call directly. The one thing that failed here was the assumption that Prometheus monitoring its own health was enough to know when Prometheus itself had stopped working.

the aftermath

Nothing about this required a traffic spike or a bad deploy to checkout-api's actual request handling. A label meant to live for an hour lived for five, and the metric it sat on multiplied it across every bucket it already had. The monitoring stack didn't fail loudly because there was nothing left inside it to raise the alarm.

The sample limit of 20,000 isn't a permanent number, it's a tripwire sized well above legitimate cardinality today so the next unbounded label gets caught on one target instead of taking down the process it's scraped by.