Every Deploy Threw a Burst of 500s: Gunicorn --preload Shared One Postgres Socket Across Four Workers
16:02 UTC. The deploy of orders-api finishes and the Sentry alert channel lights up
inside a minute: psycopg2.OperationalError: SSL error: decryption failed or bad record
mac. Eleven events, then thirty, then sixty. The 5xx graph shows a spike the width of a
coffee break, and by 16:04 it is flat again.
the scramble
This is the third deploy in a row that has done this. Nobody had connected them until now, because each burst cleared itself before anyone finished opening the dashboard. On-call pulls the last three: 0.4% of requests failed in the first ninety seconds after each rollout, then nothing for the rest of the pod's life.
First theory: the RDS instance. A bad record MAC is a TLS integrity failure, and the RDS CA certificate had been rotated the previous week. The team checked the certificate bundle in the image. It was current, and a pod that had been up for six hours connected to the same database without a single TLS error. A bad bundle would fail every connection, not a ninety-second window.
Second theory: the network. A flaky NAT or a middlebox mangling packets would corrupt TLS records and produce exactly this message. But the failures only happened on freshly started pods, on every node, in every availability zone. Middleboxes do not know how old a pod is.
the hunt
The pattern that mattered was pod age. The errors clustered in the first minute and a half and then stopped for good. Something about startup state was wrong and then healed itself. The deploy history had one change that touched startup: nine days earlier the container command had gone from a plain Gunicorn invocation to this.
--preload imports the application once in the master process and then forks the
workers, so they share the imported code pages copy-on-write. It saved about 880MB per pod, which
is why someone added it. It also means any module-level side effect now runs once, in the master,
before the fork. The team grepped for module-level work in the app package and found this:
A feature flag table loaded once at import. That query opens a real TLS connection to Postgres in the master process, runs, and returns the connection to the pool. The master then forks four workers. Each worker gets a copy of the pool, and the pool contains one idle connection whose socket file descriptor now refers to the same kernel socket in all five processes.
This explained the shape of the failures, but it was still a theory. The first attempt to prove it
was a dead end: the team compared connection counts in pg_stat_activity and saw
nothing odd, because one shared connection looks like one connection. What worked was asking each
worker which backend it was talking to on its first request:
Four workers, four different OS pids, one Postgres backend pid: 20817. All four
were using the same server connection.
the find
A TLS session is stateful. Each side keeps a sequence number and cipher state, and every record
is authenticated against it. After the fork, each worker has its own copy of that state in its
own memory but shares the socket. Worker A sends a query and advances its copy of the write
sequence number. Worker B sends a query on the same socket using the same stale state, so the
server sees a record with the wrong sequence number and fails the MAC check. Or A reads the
response that B was waiting for, and its own state cannot decrypt it. Either way you get
decryption failed or bad record mac.
It stopped after ninety seconds because SQLAlchemy's pool does the right thing once a connection errors out. The failing request invalidates the connection, the pool discards it, and the next checkout opens a fresh one owned by a single process. Four workers each burned the inherited connection on their first few requests, and after that the problem was gone. It also explains why the burst scaled with traffic during rollout: more concurrent first requests, more collisions.
the fix
SQLAlchemy documents this case. A pool must not cross a fork, and the fix is to discard the parent's connections in the child without closing them, since closing would send a TLS shutdown down the shared socket and kill the master's session for everyone. Gunicorn has a hook for it:
That is enough for correctness. The flag load also should not hold a connection in the master at all, so the second change was to stop leaving one behind:
They kept both. The hook covers the next person who adds a module-level query. The throwaway engine means the flag load never leaves a connection behind to begin with.
the aftermath
- The next deploy produced zero SSL errors across 4 workers and 6 pods. Over the nine days before the fix, three deploys had failed about 1,900 requests combined.
- Memory stayed at the lower number: roughly 190MB private per worker with
--preloadstill on. Removing it was never the right answer. - A CI check now boots the app with
--preloadand asserts thatengine.pool.checkedin()is 0 before the fork. It fails the build if import-time code leaves a connection behind. - The runbook has a new line under Postgres errors: if an SSL or protocol error only appears
right after startup, compare
pg_backend_pid()across workers before suspecting the database or the network.
We spent most of an afternoon blaming RDS and the network. The error message was honest, though: the integrity check really did fail, because two processes were writing to one encrypted stream. Nothing in the message says that, and I would not have guessed it without the pid comparison.