September 18, 2026Database8 min read

How PgBouncer's Cached DNS Sent Writes to a Demoted Aurora Primary for Six Minutes

Published September 18, 20268 min read

02:47 UTC. A routine RDS maintenance window kicks off an Aurora Postgres minor-version upgrade on the writer instance. Thirty seconds later, error rates on every write endpoint spike to 40%. The failover Aurora just performed completed in under a second, exactly as designed. The app didn't notice for six more minutes.

the setup

The stack: a Node/Express API talking to Aurora PostgreSQL through PgBouncer in transaction pooling mode, sitting on its own small EC2 box between the app tier and the database. PgBouncer was added eighteen months earlier for exactly the reason most teams add it: Aurora has a hard connection ceiling, the app runs on autoscaled ECS tasks that each want their own pool, and pooling at a single choke point kept max_connections from ever becoming a page.

pgbouncer.ini, the relevant lines

[databases]
app_db = host=app-cluster.cluster-abc123xyz.us-east-1.rds.amazonaws.com port=5432 dbname=app

[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 40
server_reset_query = DISCARD ALL

That host= line is a hostname, not an IP, and it points at Aurora's cluster writer endpoint: a CNAME that Amazon deliberately publishes with a 5-second TTL. The short TTL exists for one reason. When Aurora fails over, whether triggered by a maintenance patch, a hardware fault, or a manual failover for testing, the writer endpoint gets repointed to whichever instance is now primary, and any client that respects DNS TTLs picks up the new address within seconds. It's the mechanism Aurora's own failover-speed marketing leans on.

the scramble

The on-call engineer's first read of the PagerDuty page was a connection pool exhaustion alert, since that's the failure mode PgBouncer had produced twice before. SHOW POOLS showed cl_waiting at zero and sv_active comfortably under default_pool_size. Not exhaustion. Connections were available; they were just failing.

Second theory: the app's own retry logic had regressed in the deploy that shipped four hours earlier. That deploy touched an unrelated notifications queue, but "unrelated deploy causes unrelated failure" has been true often enough on this team that it got fifteen minutes of investigation before the timing made it implausible: the notifications deploy had been stable for four hours, and the error spike started at the exact second CloudWatch logged the Aurora failover event.

Third theory, floated and initially dismissed because "the failover already finished": maybe the app was still talking to the old primary somehow. It felt wrong on its face because Aurora failovers are sub-second and RDS Event Notifications had already confirmed the new writer was live and accepting connections. The instinct that a completed failover meant a completed transition turned out to be exactly the assumption worth checking.

the hunt

The actual write errors, once someone read past the generic 500s in the app logs to the Postgres error text underneath, were specific:

application logs, 02:47:34 UTC

error: cannot execute INSERT in a read-only transaction
    at Parser.parseErrorMessage (pg-protocol/parser.js:283)
  detail: null
  code: '25006'

Postgres error code 25006 is unambiguous: the session is connected to a database that's in read-only mode. Aurora demotes the old writer to a read replica as part of failover. If the app was hitting 25006, some fraction of its write traffic was still landing on the demoted instance, not the new one, three minutes after Aurora's own event log said the new writer was up.

That pointed straight at PgBouncer, since it's the only layer between the app and Aurora that holds long-lived state about which host to connect to. SHOW SERVERS in PgBouncer's admin console lists every backend connection it currently has open, including the resolved IP:

psql -p 6432 pgbouncer -c "SHOW SERVERS"

       host        | port |  state  |    connect_time
--------------------+------+---------+---------------------
 10.0.14.22         | 5432 | active  | 2026-09-18 02:12:09
 10.0.14.22         | 5432 | idle    | 2026-09-18 02:31:44
 10.0.14.22         | 5432 | active  | 2026-09-18 02:44:02

Every server connection PgBouncer held was pointed at 10.0.14.22. A fresh dig against the cluster writer endpoint from the same box told a different story:

dig +short app-cluster.cluster-abc123xyz.us-east-1.rds.amazonaws.com

10.0.31.9

Two different IPs. DNS already knew the writer had moved. PgBouncer didn't, because PgBouncer doesn't re-resolve a backend hostname on any schedule of its own. It resolves the address once, when a database entry is first used or when it's told to via RELOAD, and then holds that resolved IP for every connection it opens afterward, regardless of what the DNS record does in the meantime. The 5-second TTL Aurora publishes to make failover fast was invisible to the one component sitting directly in the write path.

10.0.14.22, the old writer, hadn't disappeared. Aurora Multi-AZ deliberately keeps a demoted primary reachable and serving reads through the transition instead of dropping connections outright, so anything doing SELECTs through PgBouncer against that stale IP kept working the entire time, which is exactly why the on-call engineer's first manual read-only sanity check ("can I even query the database right now") came back clean and briefly pointed the investigation away from PgBouncer instead of toward it.

the find

Root cause: PgBouncer caches the resolved IP for a backend hostname indefinitely and does not respect the DNS record's TTL. Aurora's writer endpoint is designed around a 5-second TTL so failover propagates to clients within seconds, but that only holds for clients that actually re-resolve on that cadence. PgBouncer isn't one of them by default. Reads kept succeeding against the demoted instance the whole time, which delayed detection, while writes failed with 25006 until something forced PgBouncer to drop its stale connections and look up the hostname again.

the fix

The immediate mitigation was a manual RELOAD, which makes PgBouncer re-resolve every configured hostname and, combined with closing the existing server connections, forces new ones onto the current IP:

manual recovery during the incident

psql -p 6432 pgbouncer -c "RELOAD"
psql -p 6432 pgbouncer -c "PAUSE app_db"
psql -p 6432 pgbouncer -c "RESUME app_db"

The durable fix was PgBouncer 1.19's dns_max_ttl setting, which caps how long a resolved address is trusted before PgBouncer re-resolves it on its own, no manual RELOAD required:

pgbouncer.ini, after

[pgbouncer]
dns_max_ttl = 5
dns_zone_check_period = 5

Setting dns_max_ttl to match Aurora's own 5-second TTL means PgBouncer's view of the writer's address can never be more stale than the DNS record it's reading from. A second, independent layer was added on top: an RDS Event Subscription filtered to failover events, delivered via SNS to a small Lambda that calls RELOAD over PgBouncer's admin socket the moment Aurora reports a failover, instead of waiting up to 5 seconds for the TTL-based re-resolution to notice on its own.

failover-reload.js, triggered by the SNS subscription

exports.handler = async (event) => {
  const message = JSON.parse(event.Records[0].Sns.Message);
  if (message.EventCategories?.includes('failover')) {
    const client = new Client({ host: PGBOUNCER_ADMIN_HOST, port: 6432, database: 'pgbouncer' });
    await client.connect();
    await client.query('RELOAD');
    await client.end();
  }
};

the aftermath

6 min From failover start to PgBouncer picking up the new writer IP

40% Write error rate during the window, reads unaffected the whole time

5s dns_max_ttl set after the incident, matching Aurora's own TTL

2 Failovers since, both under 8 seconds of write errors

A synthetic check now runs a write through PgBouncer every 30 seconds from outside the cluster and alerts on 25006 by itself, not just on generic 5xx rates, since that error code is the one signal that distinguishes "database down" from "pool talking to the wrong host."

A short DNS TTL only protects you if every layer between the client and the database actually re-resolves on that schedule. Connection poolers are exactly the kind of thing likely to cache past it, because caching a resolved address is usually the right call for performance, right up until the address changes underneath it.
Reads succeeding is not evidence that a failover fully propagated. Aurora keeps a demoted primary reachable for reads during the transition to avoid a hard cutover, and that same design choice is what made this incident hard to spot.
"The failover already completed" and "every client already knows about it" are two different claims. RDS Event Notifications confirm the former. Nothing confirms the latter unless you build the check yourself.
When a component's own logs can be diffed directly against ground truth, reach for that before broader theories about the app or the deploy pipeline. SHOW SERVERS against a fresh dig turned "something's stale somewhere" into a provable claim in under a minute.

PgBouncer had been sitting in the write path for eighteen months without a single DNS-related incident, because in eighteen months the writer endpoint had never had to move while PgBouncer was holding an open connection to it. The failover mechanism Aurora built to make this fast was never the weak link. The thing in front of it that never thought to ask again was.