Postgres Went Read-Only at 03:12: One Docker json-file Log Had Eaten 71GB of the Host Disk
03:12 UTC. PagerDuty fires checkout-api 5xx > 20%, and thirty seconds later a
second page arrives for the same service: write latency p99 > 10s. The Sentry
feed is one repeating line:
psycopg2.errors.DiskFull: could not extend file "base/16384/24871": No space left on
device. Reads are fine. Every INSERT and UPDATE is failing.
the scramble
Priya is on call. She opens the incident channel at 03:14 and pastes the Sentry line. The setup is plain: one 8 vCPU box running Docker Compose with the API, a worker, Redis, and Postgres 16 in a container with its data directory on a bind mount. The root volume is a 120GB EBS disk.
First theory, from Priya and from Marcus who joined at 03:16: Postgres ate its own disk. The usual suspects are WAL piling up behind a stuck replication slot, or a runaway query spilling temp files. Both have happened to this team before. Priya runs the cheap check first.
The database directory is 38GB, which is about what it was last month. The disk is 120GB full. Something else owns 80GB.
the hunt
Dead end one: the WAL. du -sh /srv/pgdata/pg_wal returned 1.1GB, and
pg_replication_slots was empty because this box has no replica. Marcus also checked
pgsql_tmp under the data directory for spilled sort files. Empty. Postgres was the
victim here, not the culprit.
Dead end two: the obvious cleanup. At 03:24 Marcus ran docker system prune -af to
recover space and bought breathing room that did not exist.
Three hundred megabytes. Postgres got a few writes through and failed again within a minute. Prune removes stopped containers, dangling images, and unused networks. It does not touch the log file of a container that is still running, and that is where the space actually was.
Priya stopped guessing and sorted by size from the top of the filesystem, one level at a time:
One file, 71GB. docker ps mapped the id to the worker container.
the find
Docker's default logging driver is json-file, and its default options are no
max-size and no max-file. Every line a container writes to stdout or
stderr is appended to a single JSON file on the host, forever, until the container is removed.
Nothing in the default configuration rotates it.
The next question was why the worker suddenly wrote that much. The git history showed it in
four minutes. Nine days earlier, during an unrelated incident with a flaky payment provider,
someone had set LOG_LEVEL=DEBUG on the worker in docker-compose.yml to
see the provider's raw responses. The incident closed. The environment variable did not get
reverted. At DEBUG the worker logged every poll of the job queue, about 400 lines a second, each
roughly 220 bytes after JSON wrapping.
The numbers fit. Nothing alerted along the way because the only disk alarm in CloudWatch watched
EBS burst balance, not filesystem usage. Nobody had ever set up the agent that reports
disk_used_percent.
the fix
Immediate recovery at 03:41, on the running container:
Postgres accepted writes again within seconds, and the 5xx rate returned to baseline by 03:43. Total write outage: 31 minutes. The API's retry queue replayed the failed checkouts, and 23 orders needed manual reconciliation because their retries had exhausted.
The permanent fix has a trap in it. The obvious move is to put rotation in the daemon config:
Marcus applied that and restarted the daemon at 04:05. Then he checked the live worker's log and it was growing past 50MB without rotating. Daemon-level defaults only apply to containers created after the change. Existing containers keep the log config they were created with, so every container on the box was still unbounded until it was recreated. The fix that actually covered the stack was also putting the limit in Compose, so it travels with the repo:
Then docker compose up -d --force-recreate during a low-traffic window, and
docker inspect --format '{{.HostConfig.LogConfig}}' <container> on each one
to confirm the options were actually present. The worst case for any container is now 150MB of
logs.
the aftermath
- Outage: 31 minutes of failed writes, 23 orders reconciled by hand, no data lost.
- Root cause chain: a debug flag outlived its incident, on a driver with no rotation, on a disk with no usage alarm. Any one of those three being fixed would have prevented the page.
- CloudWatch agent now reports
disk_used_percentfor/, with a warning at 70% and a page at 85%. At the observed 7.6GB per day, 70% would have fired about three days before the outage. - The compose file's
LOG_LEVELnow defaults to INFO, and a CI check fails the build if the string DEBUG appears in a committed compose file. - The incident template got one new checklist line: revert any temporary flags set during this incident, and name who verified it.
The disk was never a Postgres problem. Postgres just happened to be the first process that needed to grow a file, so it was the first one to say so.