Postgres Went Read-Only at 03:12: One Docker json-file Log Had Eaten 71GB of the Host Disk
← Back
November 15, 2026Docker8 min read

Postgres Went Read-Only at 03:12: One Docker json-file Log Had Eaten 71GB of the Host Disk

Published November 15, 20268 min read

03:12 UTC. PagerDuty fires checkout-api 5xx > 20%, and thirty seconds later a second page arrives for the same service: write latency p99 > 10s. The Sentry feed is one repeating line: psycopg2.errors.DiskFull: could not extend file "base/16384/24871": No space left on device. Reads are fine. Every INSERT and UPDATE is failing.


the scramble

Priya is on call. She opens the incident channel at 03:14 and pastes the Sentry line. The setup is plain: one 8 vCPU box running Docker Compose with the API, a worker, Redis, and Postgres 16 in a container with its data directory on a bind mount. The root volume is a 120GB EBS disk.

First theory, from Priya and from Marcus who joined at 03:16: Postgres ate its own disk. The usual suspects are WAL piling up behind a stuck replication slot, or a runaway query spilling temp files. Both have happened to this team before. Priya runs the cheap check first.

$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p1 120G 120G 0 100% / $ du -sh /srv/pgdata 38G /srv/pgdata

The database directory is 38GB, which is about what it was last month. The disk is 120GB full. Something else owns 80GB.


the hunt

Dead end one: the WAL. du -sh /srv/pgdata/pg_wal returned 1.1GB, and pg_replication_slots was empty because this box has no replica. Marcus also checked pgsql_tmp under the data directory for spilled sort files. Empty. Postgres was the victim here, not the culprit.

Dead end two: the obvious cleanup. At 03:24 Marcus ran docker system prune -af to recover space and bought breathing room that did not exist.

$ docker system prune -af ... Total reclaimed space: 312MB

Three hundred megabytes. Postgres got a few writes through and failed again within a minute. Prune removes stopped containers, dangling images, and unused networks. It does not touch the log file of a container that is still running, and that is where the space actually was.

Priya stopped guessing and sorted by size from the top of the filesystem, one level at a time:

$ du -xh / --max-depth=1 2>/dev/null | sort -rh | head -4 120G / 75G /var 38G /srv $ du -xh /var/lib/docker --max-depth=3 2>/dev/null | sort -rh | head -3 73G /var/lib/docker 71G /var/lib/docker/containers 71G /var/lib/docker/containers/9f3c1e07b2a4... $ ls -lh /var/lib/docker/containers/9f3c1e07b2a4*/*-json.log -rw-r----- 1 root root 71G Nov 15 03:27 9f3c1e07b2a4...-json.log

One file, 71GB. docker ps mapped the id to the worker container.


the find

Docker's default logging driver is json-file, and its default options are no max-size and no max-file. Every line a container writes to stdout or stderr is appended to a single JSON file on the host, forever, until the container is removed. Nothing in the default configuration rotates it.

The next question was why the worker suddenly wrote that much. The git history showed it in four minutes. Nine days earlier, during an unrelated incident with a flaky payment provider, someone had set LOG_LEVEL=DEBUG on the worker in docker-compose.yml to see the provider's raw responses. The incident closed. The environment variable did not get reverted. At DEBUG the worker logged every poll of the job queue, about 400 lines a second, each roughly 220 bytes after JSON wrapping.

400 lines/s x 220 bytes x 86,400 s = about 7.6GB per day 7.6GB x 9.4 days = 71GB

The numbers fit. Nothing alerted along the way because the only disk alarm in CloudWatch watched EBS burst balance, not filesystem usage. Nobody had ever set up the agent that reports disk_used_percent.

the fix

Immediate recovery at 03:41, on the running container:

# truncate in place; do NOT rm, dockerd keeps the fd open and the space stays allocated $ truncate -s 0 /var/lib/docker/containers/9f3c1e07b2a4*/*-json.log $ df -h / /dev/nvme0n1p1 120G 49G 71G 41% /

Postgres accepted writes again within seconds, and the 5xx rate returned to baseline by 03:43. Total write outage: 31 minutes. The API's retry queue replayed the failed checkouts, and 23 orders needed manual reconciliation because their retries had exhausted.

The permanent fix has a trap in it. The obvious move is to put rotation in the daemon config:

# /etc/docker/daemon.json { "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "3" } }

Marcus applied that and restarted the daemon at 04:05. Then he checked the live worker's log and it was growing past 50MB without rotating. Daemon-level defaults only apply to containers created after the change. Existing containers keep the log config they were created with, so every container on the box was still unbounded until it was recreated. The fix that actually covered the stack was also putting the limit in Compose, so it travels with the repo:

# docker-compose.yml x-logging: &default-logging driver: json-file options: max-size: "50m" max-file: "3" services: worker: logging: *default-logging environment: LOG_LEVEL: ${LOG_LEVEL:-INFO} api: logging: *default-logging postgres: logging: *default-logging

Then docker compose up -d --force-recreate during a low-traffic window, and docker inspect --format '{{.HostConfig.LogConfig}}' <container> on each one to confirm the options were actually present. The worst case for any container is now 150MB of logs.

the aftermath

  • Outage: 31 minutes of failed writes, 23 orders reconciled by hand, no data lost.
  • Root cause chain: a debug flag outlived its incident, on a driver with no rotation, on a disk with no usage alarm. Any one of those three being fixed would have prevented the page.
  • CloudWatch agent now reports disk_used_percent for /, with a warning at 70% and a page at 85%. At the observed 7.6GB per day, 70% would have fired about three days before the outage.
  • The compose file's LOG_LEVEL now defaults to INFO, and a CI check fails the build if the string DEBUG appears in a committed compose file.
  • The incident template got one new checklist line: revert any temporary flags set during this incident, and name who verified it.

The disk was never a Postgres problem. Postgres just happened to be the first process that needed to grow a file, so it was the first one to say so.

Share this
← All Posts8 min read