How a Silent Panic in Our Vault Lease Renewer Killed 412 Live Postgres Connections at Once

02:47 UTC. PagerDuty pages orders-api and payments-api within the same six seconds. Both show the same shape: error rate jumps from 0.1% to 94% in under ten seconds, and the logs filling up are all some flavor of "password authentication failed" or "terminating connection due to administrator command." Nothing crashed. Nothing deployed in the last four hours. Every connection just stopped being valid, all at once, like someone pulled a plug that doesn't exist on any diagram.

the scramble

First theory, the one that gets a security engineer paged alongside the on-call: a credential leak. A flood of simultaneous auth failures against the same handful of Postgres roles looks exactly like what a brute-force script or a leaked password would produce. Someone starts pulling the last 24 hours of IAM activity and VPC flow logs, bracing for a bad morning.

That theory dies within five minutes. Every failing connection attempt originates from an app server IP already on the allowlist, nowhere external, no new source showing up anywhere in the flow logs. Whatever's rejecting these connections isn't reacting to a stranger. It's rejecting its own fleet.

Second theory: Postgres itself is in trouble, out of memory, hit max_connections, mid-failover. The RDS console says otherwise, CPU flat, memory flat, no failover event, no restart. The primary is healthy and answering every query it's asked. It just isn't willing to answer them from the credentials it was answering them from ninety seconds ago.

the hunt

Both services pull Postgres credentials dynamically from HashiCorp Vault's database secrets engine, a decision made over a year earlier specifically to get rid of long-lived passwords sitting in an env var. Each service is issued a role scoped to it, a one-hour TTL, renewed well before expiry by a background goroutine so the pool never has to reconnect mid-traffic:

func (r *Renewer) renewLoop(ctx context.Context) {
  defer func() {
    if err := recover(); err != nil {
      log.Printf("renewer panic recovered: %v", err) // logs to a buffer nobody reads
    }
  }()

  ticker := time.NewTicker(15 * time.Minute)
  for range ticker.C {
    resp, err := r.client.Sys().Renew(r.leaseID, 0)
    if err != nil {
      log.Printf("renew failed: %v", err)
      continue
    }
    r.leaseID = resp.Secret.LeaseID // panics if resp.Secret is nil
    r.ttl = resp.Secret.LeaseDuration
  }
}

Grepping the renewer's own output narrows it fast:

$ grep -h "renew" orders-api-renewer.log | tail -5
2026-11-01T01:32:04Z renew succeeded, new ttl 3600s
2026-11-01T01:47:04Z renew succeeded, new ttl 3600s
2026-11-01T02:02:04Z renewer panic recovered: runtime error: invalid memory address
2026-11-01T02:17:04Z renewer panic recovered: runtime error: invalid memory address
2026-11-01T02:32:04Z renewer panic recovered: runtime error: invalid memory address

Three consecutive renew attempts crashed the goroutine, and the recover() swallowed each one into a log line nobody alerts on. The last successful renewal was at 01:47:04 UTC, exactly one hour before the lease's one-hour TTL ran out at 02:47:04. That's not a coincidence, that's the TTL doing exactly what a TTL does when nothing renews it in time.

What made resp.Secret nil on three straight calls was a Vault cluster upgrade that had started at 01:58 UTC, a routine rolling restart across Vault's nodes. During leader re-election, a renew call against a follower briefly returns a response with only a warnings field and no Secret, a documented behavior for a handful of Vault API calls mid-election that the renewer's code had never accounted for. It assumed resp.Secret would always be populated on a non-error response and indexed straight into it.

That explains why renewal stopped. It doesn't yet explain why the lease expiring killed live connections instead of just blocking new ones, which is what most teams would assume a TTL expiry does. The Vault audit log answers that:

{"time":"2026-11-01T02:47:02Z","type":"response","auth":{"display_name":"orders-api"},
 "request":{"operation":"revoke-lease","path":"database/revoke/orders-rw"},
 "response":{"data":{"revocation_statements":[
   "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = '{{name}}';",
   "REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM \"{{name}}\";",
   "DROP ROLE IF EXISTS \"{{name}}\";"
 ]}}}

That pg_terminate_backend line isn't Vault's out-of-the-box default. It was added to this role's custom revocation_statements eight months earlier, after a smaller incident where expired Vault roles kept failing to drop because Postgres refuses DROP ROLE on a role that still owns open sessions, leaving a slow accumulation of orphaned roles in pg_roles. The fix back then was correct for the problem it solved. It also meant that from that point on, every lease revocation, planned or not, would forcibly kill every live session under that role before dropping it. Nobody revisited that decision once the orphaned-role problem went away.

the find

A Vault rolling upgrade triggered a brief, documented response shape the renewer's code didn't handle. That panicked a goroutine whose recover() hid the failure from every alert that existed. One hour later, with no renewal having succeeded, Vault's own expiration manager did exactly what a TTL expiry is supposed to do and revoked the lease. The revocation statements it ran had been quietly carrying a pg_terminate_backend call since a previous, unrelated fix. Orders-api's pool of 250 and payments-api's pool of 162 both sat on the same Vault role, so all 412 connections went down in the same instant.

the fix

Three changes, in order of how fast they shipped. First, the renewer stops trusting that a successful API call implies a populated response, and falls back to requesting a brand-new credential set instead of crashing:

resp, err := r.client.Sys().Renew(r.leaseID, 0)
if err != nil || resp == nil || resp.Secret == nil {
  log.Printf("renew unusable, requesting fresh credentials: err=%v", err)
  renewFailures.Inc() // now a real metric, not a buried log line
  if err := r.fetchNewCredentials(ctx); err != nil {
    log.Printf("fresh credential fetch also failed: %v", err)
  }
  continue
}
r.leaseID = resp.Secret.LeaseID
r.ttl = resp.Secret.LeaseDuration

Second, a Prometheus gauge exports seconds-until-lease-expiry per credential set, independent of whether the renewer thinks it's healthy. An alert fires if any credential set drops under 10 minutes of remaining TTL, which would have paged someone at roughly 02:37 UTC, ten minutes before the actual outage, instead of relying on the renewer to accurately report its own failure.

Third, and the one that actually changes the blast radius: orders-api and payments-api no longer share a Vault role. Splitting them into per-service roles means a single bad lease can now only take down one service's connection pool, not both. The pg_terminate_backend revocation statement stays, it's still the right fix for the orphaned-role problem it was written for, but it no longer gets to detonate across two services from one shared lease.

the aftermath

Checkout error rate held above 90% for the first four minutes, until pods began individually detecting dead connections and triggering fetchNewCredentials on their own, independent of the still-broken renewer. No partial writes, no corrupted orders, every failed request simply failed to connect and either retried client-side or surfaced an error. 1,140 checkout attempts failed outright in that window; about two-thirds retried successfully once connections recovered, the rest were abandoned carts with no way to recover the lost conversions after the fact.

Dynamic secrets did their job, nothing leaked and nothing stayed valid longer than it should have. The part nobody had priced in was what happens to the connections still standing on top of a credential the instant it's taken away.