Every Deploy Threw a Burst of 500s: Gunicorn --preload Shared One Postgres Socket Across Four Workers
← Back
November 14, 2026Python8 min read

Every Deploy Threw a Burst of 500s: Gunicorn --preload Shared One Postgres Socket Across Four Workers

Published November 14, 20268 min read

16:02 UTC. The deploy of orders-api finishes and the Sentry alert channel lights up inside a minute: psycopg2.OperationalError: SSL error: decryption failed or bad record mac. Eleven events, then thirty, then sixty. The 5xx graph shows a spike the width of a coffee break, and by 16:04 it is flat again.


the scramble

This is the third deploy in a row that has done this. Nobody had connected them until now, because each burst cleared itself before anyone finished opening the dashboard. On-call pulls the last three: 0.4% of requests failed in the first ninety seconds after each rollout, then nothing for the rest of the pod's life.

First theory: the RDS instance. A bad record MAC is a TLS integrity failure, and the RDS CA certificate had been rotated the previous week. The team checked the certificate bundle in the image. It was current, and a pod that had been up for six hours connected to the same database without a single TLS error. A bad bundle would fail every connection, not a ninety-second window.

Second theory: the network. A flaky NAT or a middlebox mangling packets would corrupt TLS records and produce exactly this message. But the failures only happened on freshly started pods, on every node, in every availability zone. Middleboxes do not know how old a pod is.


the hunt

The pattern that mattered was pod age. The errors clustered in the first minute and a half and then stopped for good. Something about startup state was wrong and then healed itself. The deploy history had one change that touched startup: nine days earlier the container command had gone from a plain Gunicorn invocation to this.

# before gunicorn app.main:app -k uvicorn.workers.UvicornWorker -w 4 -b 0.0.0.0:8000 # after (memory: 4 x 410MB RSS down to roughly 4 x 190MB private) gunicorn app.main:app -k uvicorn.workers.UvicornWorker -w 4 --preload -b 0.0.0.0:8000

--preload imports the application once in the master process and then forks the workers, so they share the imported code pages copy-on-write. It saved about 880MB per pod, which is why someone added it. It also means any module-level side effect now runs once, in the master, before the fork. The team grepped for module-level work in the app package and found this:

# app/db.py engine = create_engine(settings.database_url, pool_size=5, pool_pre_ping=True) # app/flags.py, imported by app/main.py at import time with engine.connect() as conn: FLAGS = dict(conn.execute(text("SELECT name, enabled FROM feature_flags")))

A feature flag table loaded once at import. That query opens a real TLS connection to Postgres in the master process, runs, and returns the connection to the pool. The master then forks four workers. Each worker gets a copy of the pool, and the pool contains one idle connection whose socket file descriptor now refers to the same kernel socket in all five processes.

This explained the shape of the failures, but it was still a theory. The first attempt to prove it was a dead end: the team compared connection counts in pg_stat_activity and saw nothing odd, because one shared connection looks like one connection. What worked was asking each worker which backend it was talking to on its first request:

@app.middleware("http") async def log_backend_pid(request, call_next): if not getattr(app.state, "pid_logged", False): app.state.pid_logged = True with engine.connect() as conn: pid = conn.execute(text("SELECT pg_backend_pid()")).scalar() logger.warning("worker=%s backend_pid=%s", os.getpid(), pid) return await call_next(request)

Four workers, four different OS pids, one Postgres backend pid: 20817. All four were using the same server connection.


the find

A TLS session is stateful. Each side keeps a sequence number and cipher state, and every record is authenticated against it. After the fork, each worker has its own copy of that state in its own memory but shares the socket. Worker A sends a query and advances its copy of the write sequence number. Worker B sends a query on the same socket using the same stale state, so the server sees a record with the wrong sequence number and fails the MAC check. Or A reads the response that B was waiting for, and its own state cannot decrypt it. Either way you get decryption failed or bad record mac.

It stopped after ninety seconds because SQLAlchemy's pool does the right thing once a connection errors out. The failing request invalidates the connection, the pool discards it, and the next checkout opens a fresh one owned by a single process. Four workers each burned the inherited connection on their first few requests, and after that the problem was gone. It also explains why the burst scaled with traffic during rollout: more concurrent first requests, more collisions.

the fix

SQLAlchemy documents this case. A pool must not cross a fork, and the fix is to discard the parent's connections in the child without closing them, since closing would send a TLS shutdown down the shared socket and kill the master's session for everyone. Gunicorn has a hook for it:

# gunicorn.conf.py def post_fork(server, worker): from app.db import engine # close=False: drop references, do NOT send a close on the shared socket engine.dispose(close=False)

That is enough for correctness. The flag load also should not hold a connection in the master at all, so the second change was to stop leaving one behind:

# app/flags.py from sqlalchemy import create_engine, text from sqlalchemy.pool import NullPool def load_flags(url): # throwaway engine, no pool, nothing survives the fork tmp = create_engine(url, poolclass=NullPool) with tmp.connect() as conn: return dict(conn.execute(text("SELECT name, enabled FROM feature_flags"))) FLAGS = load_flags(settings.database_url)

They kept both. The hook covers the next person who adds a module-level query. The throwaway engine means the flag load never leaves a connection behind to begin with.

the aftermath

  • The next deploy produced zero SSL errors across 4 workers and 6 pods. Over the nine days before the fix, three deploys had failed about 1,900 requests combined.
  • Memory stayed at the lower number: roughly 190MB private per worker with --preload still on. Removing it was never the right answer.
  • A CI check now boots the app with --preload and asserts that engine.pool.checkedin() is 0 before the fork. It fails the build if import-time code leaves a connection behind.
  • The runbook has a new line under Postgres errors: if an SSL or protocol error only appears right after startup, compare pg_backend_pid() across workers before suspecting the database or the network.

We spent most of an afternoon blaming RDS and the network. The error message was honest, though: the integrity check really did fail, because two processes were writing to one encrypted stream. Nothing in the message says that, and I would not have guessed it without the pid comparison.

Share this
← All Posts8 min read