How an Orphaned Replication Slot Filled Our Postgres Disk and Stopped Every Write

14:32 UTC. PagerDuty: "primary-db-01 disk utilization 91%, climbing." No deploy in the last twelve hours, no traffic spike on the dashboards. A volume just filling up, for no reason anyone on the page could name yet.

the setup

primary-db-01 is a single Postgres 16 primary on a 500GB EBS volume, backing checkout-svc, orders-svc, and a handful of smaller internal services. Two logical replication slots read off it in normal operation: checkout_cdc_prod, which feeds a Debezium connector into Kafka for the data warehouse, and until sixteen days earlier, catalog_search_sync, which fed a homegrown consumer that kept our OpenSearch catalog index in sync with the products table.

catalog-search-sync was retired when catalog indexing moved onto the same Debezium/Kafka Connect pipeline checkout already used, one topic, one connector, one thing to operate instead of two. The old consumer's deployment, service, and secrets were deleted the same day. The decommission ticket was closed as complete. Nobody ran the one SQL statement that actually mattered.

the scramble

On-call's first theory was ordinary growth: a busy week, more orders, more rows, more WAL. SELECT pg_size_pretty(pg_database_size('orders')); came back at 38GB, a couple hundred megabytes over last week. That didn't explain a volume climbing toward full.

Second theory: a runaway batch job, someone's backfill script left running against production. pg_stat_activity showed nothing unusual, no long-running transactions, no query holding a lock for hours, no obvious offender.

Third theory: autovacuum falling behind, dead tuples piling up faster than they were being reclaimed. pg_stat_user_tables showed dead tuple counts within normal range on every table that mattered. Three theories, three dead ends, twenty-eight minutes gone, and the disk kept climbing.

At 15:04 UTC the volume hit 100% and the theories stopped mattering. Writes started failing outright.

ERROR:  could not extend file "base/16391/24887": No space left on device
HINT:  Check free disk space.
STATEMENT:  INSERT INTO orders (customer_id, total_cents, status) VALUES ($1, $2, $3)

the hunt

With writes actually failing now, the next question was where the disk had gone, not what the database's own size accounting said. A raw directory comparison on the data volume answered that in one command.

$ du -sh /var/lib/postgresql/16/main/pg_wal
341G    /var/lib/postgresql/16/main/pg_wal

$ du -sh /var/lib/postgresql/16/main/base
38G     /var/lib/postgresql/16/main/base

38GB of actual data, 341GB of WAL. A healthy primary with active archiving recycles WAL segments continuously; it doesn't accumulate 300GB more of them than the database it's protecting. Something was holding every segment back from being reused. Replication slots are the usual reason, so that's what got queried next.

SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

     slot_name       | active | retained_wal
----------------------+--------+--------------
 catalog_search_sync  | f      | 341 GB
 checkout_cdc_prod    | t      | 890 MB

checkout_cdc_prod was active and healthy, retaining well under a gigabyte, exactly what you'd expect from a connector that's actually consuming. catalog_search_sync was active = f, nothing had connected to it in a very long time, and it was pinning 341GB of WAL that Postgres could not recycle because that slot's restart_lsn hadn't advanced since whenever it last had a reader.

the find

A quick check against the deploy history confirmed the timeline.

$ git log --oneline -- deploy/catalog-search-sync/
a3f21c9 chore: remove catalog-search-sync deployment (superseded by catalog-cdc)

$ git log -1 --format=%cd a3f21c9
Thu Aug 11 09:42:11 2026 +0000

Sixteen days earlier. The decommission ticket covered every application resource the old consumer owned, its deployment, its service account, its secrets, its dashboards, but the replication slot lived in Postgres itself, not in Kubernetes, so it wasn't in scope for anything the checklist actually checked. Postgres doesn't expire a logical slot on its own. It assumes a reader is coming back, and keeps every byte of WAL that reader would need, for as long as the slot exists, with no default limit on how much that can add up to.

Root cause: an orphaned logical replication slot from a decommissioned consumer, still present in Postgres sixteen days after the service that owned it was deleted, silently pinning WAL retention with no cap, until the volume it was consuming ran out of room and every write on the primary started failing.

the fix

Immediate mitigation: drop the stale slot and force a checkpoint so Postgres could actually reclaim the segments it had been holding.

SELECT pg_drop_replication_slot('catalog_search_sync');
CHECKPOINT;

Dropping the slot doesn't free the disk by itself, Postgres still waits for the next checkpoint to actually recycle the now-unpinned segments. Disk usage dropped from 97% to 21% within about ninety seconds of the checkpoint completing, and writes started succeeding again immediately after.

That stopped the bleeding but not the next occurrence. Two structural changes followed. First, a hard cap on how much WAL any single slot is allowed to retain, so a stuck or forgotten slot gets invalidated instead of filling the volume:

max_slot_wal_keep_size = 50GB

Past that limit, Postgres marks the offending slot invalid rather than continuing to retain WAL for it indefinitely. The active checkout_cdc_prod connector, which typically retains under a gigabyte, has enormous headroom under that cap. Anything approaching it now means something is actually wrong with a real consumer, worth paging on its own.

Second, a CI check added to the decommission process itself, so dropping a slot isn't a step someone has to remember:

#!/bin/bash
# Run as part of every service decommission PR. Fails the build if the
# service being removed still owns a replication slot in the primary.
SERVICE="$1"
psql "$DATABASE_URL" -tAc \
  "SELECT slot_name FROM pg_replication_slots WHERE slot_name = '${SERVICE}_sync';" \
  | grep -q . && {
    echo "FATAL: replication slot ${SERVICE}_sync still exists, drop it before merging"
    exit 1
  }
exit 0

A Datadog monitor now also watches retained WAL per slot directly, independent of overall disk usage, so a slot creeping toward its cap shows up days before the volume does.

the aftermath

Every failed write was queued and retried client-side once the volume recovered, no orders were lost, but checkout latency spiked hard for those twenty-one minutes and support fielded a run of "is checkout down" reports before the fix landed.

checkout_cdc_prod never stopped consuming through any of this. It was never the problem. It just happened to share a disk with a slot that had outlived the thing it was created for.