How a New Field in Our Flag Service's Schema Silently Froze a Checkout Rollout at 10% for Six Hours

14:02 UTC. #eng-incidents: "Third report this hour of an item vanishing from someone's cart at checkout. Didn't we ship the fix for this at 08:00?" The flag dashboard said new-cart-serialization: 100%, updated six hours earlier. On paper, the bug had been dead since breakfast.

the setup

checkout-api runs 24 pods behind a load balancer with no session affinity. Cart state travels between steps as a signed token, not a server session, which is deliberate: any pod can handle any step of a checkout for the same reason any pod can handle any request. A known bug had been quietly costing us orders for weeks. The old cart serializer wrote each line item as a separate key, and under specific interleavings of "add item" and "apply discount" requests, the last item added could get silently dropped before payment. The fix was a rewrite to serialize the whole cart as one atomic JSON blob, gated behind a flag: new-cart-serialization, rolled out gradually through our internal flag service.

Every pod runs a background thread that polls the flag service every 30 seconds (plus a few seconds of jitter, so all 24 pods don't hit it at once) and caches the result in memory. The flag itself is re-evaluated fresh on every request rather than pinned once per checkout session, which is normally harmless because canary percentages change slowly. It stopped being harmless the moment the cached value on every pod stopped changing at all.

the scramble

First theory: the fix itself had an edge case. Carts with more than ten items, carts with two stacked discount codes, carts touched from two browser tabs. Nobody could reproduce it locally against the new serializer, and the reports kept coming in at a steady trickle, not the shape of a rare edge case.

Second theory: the frontend was serving a stale bundle from cache, still running the old add-to-cart logic client-side. A quick check of response headers ruled that out in under five minutes: cache-control: no-store on the JS bundle, and the deployed asset hash matched what had shipped that morning. The frontend was fine. Whatever was wrong lived somewhere between the flag check and the serializer, and nobody had looked there yet because the dashboard said that path had been at 100% since 08:00.

the hunt

Someone added a debug route that dumps a pod's in-memory flag cache and the timestamp of its last successful refresh, then fanned it out across the fleet.

$ for i in $(seq 1 24); do
    curl -s http://checkout-api-$i.internal:8080/internal/flag-state | jq -c .
  done

{"flag":"new-cart-serialization","value":"10%","lastRefreshedAt":"2026-09-06T08:00:03Z"}
{"flag":"new-cart-serialization","value":"10%","lastRefreshedAt":"2026-09-06T08:00:07Z"}
{"flag":"new-cart-serialization","value":"10%","lastRefreshedAt":"2026-09-06T08:00:11Z"}
... (21 more, all "10%", all lastRefreshedAt between 08:00:03 and 08:00:29)

Every pod was stuck at the old 10% canary value. Every pod's last successful refresh landed within the same 26-second window, which lined up exactly with the moment the rollout bump to 100% had gone out. Not a slow drift, not a partial rollout. Every background refresh thread in the fleet had done exactly one more successful cycle after the bump, then nothing, ever again.

Pod stdout logs from that window had the answer, buried in normal INFO-level noise because nothing paged on it:

INFO  flag_refresher: fetched config, 412 bytes
Exception in thread FlagRefresher:
Traceback (most recent call last):
  File "flag_client/refresher.py", line 41, in run
    config = FlagConfig(**payload)
TypeError: __init__() got an unexpected keyword argument 'rolloutStrategy'

Days earlier, the flag service's platform team had shipped support for a future feature, per-attribute sticky bucketing, and added a rolloutStrategy field to the config schema. No existing flag actually used it yet, so no client had ever seen it in a real payload. The 100% bump on new-cart-serialization was the first rollout change to set that field, because sticky bucketing was the whole reason this flag was finally going to 100% instead of sitting at a partial canary forever. The moment that payload reached a pod, its refresh thread tried to construct a dataclass from it.

@dataclass
class FlagConfig:
    name: str
    value: str
    updatedAt: str

class FlagRefresher(threading.Thread):
    def run(self):
        while True:
            try:
                payload = fetch_flag_config()  # network call, can raise
            except RequestException:
                log.debug("flag fetch failed, keeping cached value")
                time.sleep(REFRESH_INTERVAL)
                continue

            config = FlagConfig(**payload)   # not inside the try block
            _cache[config.name] = config
            time.sleep(REFRESH_INTERVAL)

The try/except only wrapped the network call. Parsing the response into FlagConfig sat outside it, so an unrecognized keyword argument raised straight out of run(). In Python, an uncaught exception in a thread doesn't crash the process, it just ends that one thread. No supervisor was restarting it. The pod's health check endpoint never touched the flag cache, so the load balancer kept treating it as perfectly healthy. Every pod died the same way within the same half-minute window, then sat there, healthy and serving traffic, permanently reporting a flag value from before the bump.

the find

Two failures had to line up. The client's config parser used strict construction with no tolerance for fields it didn't recognize, so a forward-compatible schema change on the server became a breaking change on every client still running the older parser. And the refresh loop's error handling covered the failure mode someone had thought about, a flaky network call, but not the one that actually happened, a payload the code didn't know how to read. Once the thread died, there was no signal anywhere that it had. Not a metric, not a log line above INFO, not a health check. The flag service's own dashboard had no way to know either: it recorded what it had sent, not what any client had confirmed applying. "100% rolled out" was a description of the control plane's intent, not a fact about the fleet.

the fix

First, the parser stopped rejecting fields it doesn't understand. A schema should be able to grow without breaking clients that haven't caught up yet.

@dataclass
class FlagConfig:
    name: str
    value: str
    updatedAt: str

    @classmethod
    def from_payload(cls, payload: dict) -> "FlagConfig":
        known = {f.name for f in fields(cls)}
        return cls(**{k: v for k, v in payload.items() if k in known})

Second, the entire refresh cycle, fetch and parse together, sits inside one try/except. A bad payload now logs an error and gets retried next interval instead of ending the thread.

class FlagRefresher(threading.Thread):
    def run(self):
        while True:
            try:
                payload = fetch_flag_config()
                config = FlagConfig.from_payload(payload)
                _cache[config.name] = config
                _last_success[config.name] = time.time()
            except Exception:
                log.exception("flag refresh cycle failed, keeping cached value")
            time.sleep(REFRESH_INTERVAL)

Third, staleness became observable instead of invisible. Every pod now exports how long it's been since each flag last refreshed successfully, and an alert fires if any pod exceeds five minutes, well before "silently stuck" can turn into "stuck for six hours."

def refresh_age_seconds(flag_name: str) -> float:
    return time.time() - _last_success.get(flag_name, 0)

# scraped by Prometheus, alert fires above 300s
flag_refresh_age_seconds.labels(flag="new-cart-serialization").set(
    refresh_age_seconds("new-cart-serialization")
)

Fourth, the flag service's own dashboard changed what "rolled out" means. It now shows the oldest lastRefreshedAt reported back by any client that's called in during the last five minutes, alongside the value it thinks it sent. A rollout is confirmed once the fleet has actually said so, not once the control plane has finished sending it.

the aftermath

No page fired during those six hours because nothing was broken in a way any existing check understood. Pods were healthy, latency was normal, error rates were flat. The only signal was a support queue slowly filling with a bug everyone believed was already fixed, which is a much slower and much less trustworthy alarm than a metric crossing a threshold.

The five-minute staleness threshold is deliberately tight against a 30-second refresh interval. Ten missed cycles in a row is already a strong signal something died, and the gap between "a thread died" and "someone found out" is the entire size of this incident.