How a Cache Key's 90-Second TTL Turned a Push Notification Into 212 Duplicate Postgres Queries

14:32:26 UTC. Second page in ninety seconds. The first said postgres-primary cpu > 95% (5m). This one said feed-api p99 > 8000ms. In #incidents, someone had already dropped a Grafana screenshot: request rate for /api/trending going straight up, no ramp, just a wall.

the setup

feed-api serves GET /api/trending, the "trending now" tab. Under the hood it's a single expensive query: a window function ranking posts by an engagement score computed over the last 24 hours, joined against a few tables, limit 50. Cold, it takes about 4.8 seconds. Nobody had ever needed to make it faster, because the result is cached in Redis under one shared key, trending:v3, for 90 seconds, and baseline traffic on that endpoint sits around 150 requests per second. The cache absorbed all of it. The query ran roughly once every ninety seconds, no matter how many requests came in behind it.

async function getTrending() {
  const cached = await redis.get('trending:v3');
  if (cached) return JSON.parse(cached);

  const rows = await computeTrending(); // ~4.8s Postgres aggregation
  await redis.set('trending:v3', JSON.stringify(rows), 'EX', 90);
  return rows;
}

At 14:31:40 UTC, growth sent a push notification to 620,000 users promoting the trending tab, deep linking straight into it. Nothing wrong with that on its own. The problem was timing: the cache key had last been populated at 14:30:11 and was set to expire at 14:31:41, one second after the push went out.

the scramble

First theory, and a reasonable one: this was just organic load. Grafana showed request rate on /api/trending jumping from a 150/s baseline to over 2,400/s within a minute of the push going out. A popular notification driving a traffic spike is not an incident by itself, it's the whole point of sending one.

On-call's first move was to give the app tier more room to breathe. feed-api pod CPU was climbing too, each pod busy holding open connections waiting on Postgres, so it read like a capacity problem at the app layer. HPA had already started scaling out on its own; on-call bumped the minimum replica count from 40 to 64 to get ahead of it.

on-call: bumping feed-api min replicas 40 -> 64, HPA is already climbing
          this should take the edge off postgres cpu once app tier catches up

It didn't take the edge off anything. Two minutes later, Postgres CPU was still pinned at 97%, and feed-api p99 latency had gotten worse, not better. More pods running the same uncoordinated code path meant more concurrent callers hitting the same cache miss, not less.

Dead end two: someone checked whether a recent deploy had introduced a slow query change to the trending path. git log --oneline -- services/feed-api/src/trending showed the last change to that code was six days old, a comment update. Nothing had shipped there since. Whatever was happening wasn't a regression in the query itself.

the hunt

Postgres told the real story. pg_stat_activity showed dozens of connections running the exact same query text, all active, all started within the same few hundred milliseconds of each other.

SELECT count(*), left(query, 60)
FROM pg_stat_activity
WHERE state = 'active'
GROUP BY left(query, 60)
ORDER BY 1 DESC;

 count |                          left
-------+------------------------------------------------------------
   212 | WITH engagement_window AS (SELECT post_id, sum(score) ...
     6 | UPDATE user_sessions SET last_seen_at = now() WHERE ...
     3 | SELECT * FROM notifications WHERE user_id = $1 ORDER ...

212 copies of the trending aggregation, running at once, against a query that individually takes 4.8 seconds. A quick check of the cache explained why.

127.0.0.1:6379> GET trending:v3
(nil)
127.0.0.1:6379> TTL trending:v3
(integer) -2

The key was gone. Every one of those 212 requests had arrived after 14:31:41, found nothing in Redis, and independently done exactly what the handler code told it to do: run the full aggregation itself and write the result back. There was no coordination between them at all, nothing that said "someone else is already doing this, wait for it."

It didn't stop after the first wave repopulated the cache, either. Users kept opening the app for another twenty minutes as the push notification trickled through delivery queues, keeping request volume elevated well above baseline. Every 90 seconds, the cache expired again, and every 90 seconds a fresh batch of concurrent requests landed in the same gap, ran the same query 100-plus times at once, and pegged Postgres again. Grafana's CPU graph for the whole incident was a sawtooth, not a single spike, one tooth per TTL cycle.

the find

Root cause: a cache-aside pattern with no request coalescing around an expensive recompute path. A single shared TTL protects you from repeated work when requests are spread out. It does nothing the moment the key expires while many requests are in flight at once. Every one of them sees an empty cache and treats itself as the one responsible for filling it. The push notification didn't cause the bug. It just supplied enough concurrent traffic, landing close enough to a TTL boundary, to turn a latent stampede risk into an actual one, and it kept re-triggering every cycle for as long as traffic stayed elevated.

Scaling the app tier made it measurably worse: more feed-api pods meant more independent request handlers hitting the same miss window, which meant more concurrent duplicate queries per TTL cycle, not fewer. The bottleneck was never app-tier capacity. It was the total absence of a lock around the one code path expensive enough to need one.

the fix

A Redis-based lock around the recompute path, so only one caller per TTL cycle actually queries Postgres. Everyone else either serves a short-lived stale copy or waits briefly and retries the read.

async function getTrending() {
  const cached = await redis.get('trending:v3');
  if (cached) return JSON.parse(cached);

  const gotLock = await redis.set('trending:v3:lock', '1', 'NX', 'PX', 6000);

  if (!gotLock) {
    const stale = await redis.get('trending:v3:stale');
    if (stale) return JSON.parse(stale);
    await sleep(200);
    return getTrending(); // bounded retry, cache is usually warm by now
  }

  try {
    const rows = await computeTrending();
    await redis.set('trending:v3', JSON.stringify(rows), 'EX', 90);
    await redis.set('trending:v3:stale', JSON.stringify(rows), 'EX', 600);
    return rows;
  } finally {
    await redis.del('trending:v3:lock');
  }
}

The stale copy, kept alive for ten minutes past the real TTL, matters as much as the lock. Without it, every request that loses the lock race just queues up waiting on the one that won, and 211 waiting requests during a 4.8-second recompute is still real load on the connection pool, just serialized instead of duplicated. With it, only the lock winner ever touches Postgres; everyone else gets a slightly-old but correct answer in under a millisecond.

A monitoring gap got closed too: the query that grouped pg_stat_activity by query text and counted duplicates, the thing that actually diagnosed this, became a scheduled check that pages if any single query text has more than 20 concurrent active copies.

the aftermath

Nothing about this needed a bug in the query itself, or a bad deploy, or a config drift. The aggregation was fine. The cache was fine, most of the time. It broke because a shared cache key with a TTL only protects you between misses that happen to be spread out, and a growth team sending a well-timed push notification is exactly the kind of event that makes misses stop being spread out.

The lock adds a small amount of complexity to one function. It replaced a failure mode that could recur every ninety seconds for as long as traffic stayed elevated, with no code change and no alert required to make it stop on its own.