How a Read Receipts Launch Burned Through 2.1 Billion IDs and Stopped Every Write

03:47 UTC. PagerDuty: "notifications-worker error rate 100%, last 5 min." Every notification delivery in the pipeline failing the same way, no partial degradation, nothing intermittent. Just off.

the setup

notifications-db-01 is a dedicated Postgres 15 instance backing notifications-svc, separate from the checkout cluster. Its busiest table, notification_events, logs one row per lifecycle event on every push notification we send, created back in 2019 with a plain id SERIAL PRIMARY KEY. Nobody revisited that choice for seven years because nobody had a reason to.

Nineteen days before the page, notifications-svc shipped read receipts. Before that launch, a notification wrote a single row: sent. After it, the same notification wrote three: sent, delivered, opened. Product measured the launch by open-rate lift. Nobody measured it by sequence consumption.

the scramble

First theory: the 03:10 UTC deploy, a batching change to how notifications-worker flushed delivery confirmations. It was the only thing that had touched the path recently. On-call rolled it back. The error rate stayed at 100%.

Second theory: disk. A full volume was the closest thing anyone had seen behave like this before. df -h on notifications-db-01 came back at 34% used, nowhere close.

Third theory: connection pool exhaustion from the read receipts traffic finally catching up with PgBouncer's pool size. SHOW POOLS; showed spare capacity in every pool. Three theories, three dead ends, eleven minutes gone, and every insert into notification_events was still failing.

psycopg2.errors.SequenceGeneratorLimitExceeded: nextval: reached maximum value
of sequence "notification_events_id_seq" (2147483647)

Nobody had read the actual error yet. They'd been reading Sentry's grouped summary, "insert failed on notification_events," and guessing at causes upstream of it instead.

the hunt

Once someone opened the raw exception, there was nothing left to guess about. A sequence had hit its ceiling. The next question was how close it had been sitting to that ceiling before tonight, and whether this was sudden or a long time coming.

SELECT seqrelid::regclass, last_value, is_called
FROM pg_sequences
WHERE sequencename = 'notification_events_id_seq';

        seqrelid          | last_value | is_called
---------------------------+------------+-----------
 notification_events_id_seq | 2147483647 |    t

Sitting exactly at 2147483647, the maximum value a Postgres integer column can hold. Not close to the ceiling, at it. A pull of the read receipts launch metrics from Datadog filled in the rest: the table had been consuming roughly 170 million IDs a month before the launch, about four months of headroom left at that rate. The moment read receipts went live, every notification started writing three rows instead of one, and the burn rate roughly tripled overnight.

$ git log --oneline -- notifications-svc/features/read-receipts/
e91a204 feat: read receipts - sent/delivered/opened event tracking

$ git log -1 --format=%cd e91a204
Fri Aug 14 16:20:03 2026 +0000

Nineteen days from that deploy to the sequence hitting its ceiling. Four months of runway, cut to nineteen days, and nobody watching the one number that would have said so.

the find

Root cause: notification_events.id was a 2019-era SERIAL column, backed by an int4 sequence capped at 2,147,483,647. The read receipts launch on August 14 tripled the table's write rate by turning every single-row notification event into three. That tripled burn rate consumed the sequence's remaining headroom in nineteen days instead of the roughly four months it would have taken at the pre-launch rate, and at 03:47 UTC on September 2 the sequence issued its last valid value. Every subsequent nextval() call raised SequenceGeneratorLimitExceeded, and every insert into the table failed the same way, immediately, with no partial window where some writes still got through.

the fix

A table backed by int4 at 2.1 billion rows can't just get ALTER COLUMN ... TYPE bigint run against it during an outage. That rewrites every row under an ACCESS EXCLUSIVE lock, and on a table this size that's hours, not minutes, of every reader and writer blocked. We needed writes working again in the next few minutes, not after a multi-hour rewrite finished.

The immediate fix leans on something most engineers never need to know about int4: the range isn't 1 to 2.1 billion, it's roughly negative 2.1 billion to positive 2.1 billion. A SERIAL sequence only ever uses the positive half by default. Nothing in notification_events had ever been assigned a negative ID, and a quick grep of notifications-svc confirmed no code path assumed IDs were always positive, no URL parsing, no client-side range checks. That cleared the sequence to keep counting into the half of its range it had never touched.

ALTER SEQUENCE notification_events_id_seq MINVALUE -2147483648;
ALTER SEQUENCE notification_events_id_seq RESTART WITH -2147483648;

Writes started succeeding again within seconds of the second statement. Nine minutes of hard failure, start to finish, from the first alert to the restart landing.

That bought runway, not a real fix, roughly another four billion IDs before the sequence would hit zero and cross back into positive territory it had already used, a collision nobody wanted to find out about the hard way. The actual fix was the bigint migration we'd been putting off since 2019, done properly this time: a new bigint column backfilled in batches, kept in sync with new writes through a trigger, then swapped in as the primary key with zero downtime.

ALTER TABLE notification_events ADD COLUMN id_bigint bigint;

CREATE OR REPLACE FUNCTION sync_notification_events_id() RETURNS trigger AS $$
BEGIN
  NEW.id_bigint := NEW.id;
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_sync_notification_events_id
  BEFORE INSERT ON notification_events
  FOR EACH ROW EXECUTE FUNCTION sync_notification_events_id();

-- backfilled in 50k-row batches against ctid ranges, off-peak, over ~11 days

CREATE UNIQUE INDEX CONCURRENTLY notification_events_id_bigint_key
  ON notification_events (id_bigint);

BEGIN;
ALTER TABLE notification_events DROP CONSTRAINT notification_events_pkey;
ALTER TABLE notification_events ADD CONSTRAINT notification_events_pkey
  PRIMARY KEY USING INDEX notification_events_id_bigint_key;
ALTER TABLE notification_events ALTER COLUMN id_bigint SET DEFAULT
  nextval('notification_events_id_bigint_seq'::regclass);
COMMIT;

Every new table in notifications-svc now defaults to bigint generated always as identity, enforced by a CI lint rule that fails a migration PR on any SERIAL or bare integer primary key. A Datadog monitor tracks last_value / 2147483647 for every remaining int4 sequence in the fleet and pages at 75% consumed, which would have caught this with well over a month to spare instead of zero minutes.

the aftermath

Queued events retried successfully once writes recovered, no notification data was lost, but delivery and open tracking for that nine-minute window undercounted, and the read receipts dashboard showed a visible dip the next morning that took a support thread to explain.

checkout-svc and the rest of the fleet never touched this table and never saw an error. notifications-db-01 was its own instance, its own sequence, its own ceiling, reached alone.