How Envoy's Panic Threshold Turned a Routine Rollout Into a 14-Minute Checkout Outage

09:47 UTC. Checkout error rate alert fires, 8% and climbing. A routine rolling restart of authz-svc, the service that calls out to fraud-check before approving a payment, went out eleven minutes earlier. Nothing in the diff touches payment logic. Nothing in the diff touches fraud-check. The on-call engineer stares at a deploy that should not be able to do this.

the scramble

First theory, the obvious one: the new build is bad. The rolling restart replaced 6 of 24 authz-svc pods before anyone noticed anything, so a bad binary is the simplest explanation. On-call rolls it back. Error rate dips to 3% within a minute, which looks like confirmation, then climbs back to 11% four minutes later. A clean rollback should not behave like that.

Second theory: the fraud-check dependency itself is degraded, and the deploy is a coincidence. Fraud-check's own dashboards show nothing, p50 and p99 both flat, error rate at baseline. Pulling up individual authz-svc pod metrics one at a time doesn't help either, most pods show normal latency and close to zero errors. If one pod were broken, it should stand out. None of them do, individually. The errors are smeared across the fleet in a way that app-level, per-pod dashboards aren't built to show.

the hunt

authz-svc runs behind Istio, one Envoy sidecar per pod, and Envoy keeps its own view of cluster health separate from whatever the app reports. That view lives on the sidecar's admin interface, not in the usual dashboards:

kubectl exec authz-svc-7 -c istio-proxy -- \
  curl -s localhost:15000/clusters | grep -E "outlier_detection|panic"

outbound|9000||authz-svc.default.svc.cluster.local::outlier_detection.ejections_active::15
outbound|9000||authz-svc.default.svc.cluster.local::outlier_detection.ejections_total::41
outbound|9000||authz-svc.default.svc.cluster.local::lb_healthy_panic::1

lb_healthy_panic::1 is the detail that reframes the entire incident. 15 of the cluster's hosts are actively ejected by outlier detection, which on a 24-pod fleet is 62%, and the panic flag means Envoy has stopped trusting that ejection state altogether. Checking the DestinationRule for authz-svc explains how ejections got that high in the first place.

outlierDetection:
  consecutiveGatewayErrors: 3
  interval: 10s
  baseEjectionTime: 30s
  maxEjectionPercent: 100   # raised from the default 10, six months ago

maxEjectionPercent was raised to 100 after an earlier incident where a single genuinely broken pod kept absorbing traffic because the default 10% cap wouldn't let outlier detection eject more than two or three hosts at a time. That change did exactly what it was meant to do, for exactly one bad pod. It also removed the ceiling that would have stopped ejections from ever climbing past Envoy's own panic threshold, which defaults to 50% of the cluster. Nobody connected those two numbers at the time, because they live in different subsystems and nothing enforces that one stays under the other.

The rolling restart's pods come up fast at the container level but take roughly 8 seconds to finish warming a connection pool to fraud-check. authz-svc's own code returns a defensive 503 if a request lands before that pool is ready, rather than queuing it. Six pods restarting inside the same rollout window means a burst of 503s landing on enough hosts, inside the same 10-second outlier interval, to trip consecutiveGatewayErrors: 3 on 15 of 24 pods at once. Healthy host percentage drops from 100% to 38% in under ten seconds. Panic triggers. The rollback didn't fail to fix anything, it just restarted the same rolling update, which recreated the same synchronized 503 burst, which tripped panic a second time.

the find

Panic mode isn't a bug, it's Envoy's deliberate fail-open behavior: when too much of a cluster looks unhealthy, assume the health signal itself is unreliable and spread load across every host rather than trust ejection data that might be lying. Usually that's the right call. It becomes the wrong call the moment one of the ejected hosts isn't a false positive.

Pod authz-svc-7 wasn't warming up. Its outbound HTTP client to fraud-check was built without an explicit MaxIdleConnsPerHost, so Go's transport default of 2 idle connections per host was in effect, and it had no context timeout on that call path either, a client added after the shared factory that normally sets both. Under the traffic resumption that followed panic mode, that pod's connection pool to fraud-check exhausted immediately and every subsequent request on it queued waiting for a free connection that never freed, for the full 30-second gRPC deadline.

Panic mode sent roughly 1 in 24 requests to that one pod for as long as panic held. Those requests didn't fail fast, they hung for 30 seconds each. That's the actual shape of the checkout error rate the alert caught, not the 503 burst, which had already self-resolved within ten seconds of each pool warming up.

the fix

Immediate: kubectl delete pod authz-svc-7. Removing the one genuinely wedged pod mattered more than anything happening in Envoy, since panic mode was going to keep routing to it regardless of ejection state.

// before: shared factory's defaults silently skipped for this one client
client := &http.Client{Transport: &http.Transport{}}

// after: explicit, matches every other outbound client in the service
client := &http.Client{
    Transport: &http.Transport{
        MaxIdleConnsPerHost: 50,
        IdleConnTimeout:     60 * time.Second,
    },
    Timeout: 2 * time.Second,
}

maxEjectionPercent went back down to 40, high enough to still eject a single bad pod outright, low enough to stay under the 50% panic threshold even if every host in a rollout trips the outlier check at once. The readiness probe changed too: it now blocks traffic until the fraud-check connection pool reports ready, instead of only checking that the process is alive, so a warming pod never takes real requests in the first place.

A new alert pages directly on lb_healthy_panic and outlier_detection.ejections_active as a ratio of cluster size, separate from the aggregate checkout error rate, since the panic flag explains an outage like this one faster than any symptom-level metric does.

the aftermath

Nothing here was a single bug. The ejection threshold was reasonable the day it was raised. The panic default is reasonable almost everywhere it applies. The missing timeout was one client out of dozens. Each piece passed review on its own. The incident only existed in the gap between them, where nobody was checking whether Envoy's two independent safety thresholds could still both hold at once.