How Redis's allkeys-lru Policy Reset Our Rate Limiter and Blew Past a Metered API's Cap

11:03 UTC. The geocoding provider's dashboard, pulled up by on-call after the first ticket: 429 Too Many Requests, account-wide. Not one customer's calls failing, all of them. The provider's own status page says green across the board. Whatever this is, it's coming from our side.

the setup

The address-enrichment feature calls a metered third-party geocoding API to turn free-text addresses into coordinates. The provider bills per call above a contracted monthly volume, so every org gets an internal cap, 5,000 geocode calls per org per day, enforced by a counter in Redis before the request ever leaves the building:

async function checkDailyCap(orgId) {
  const key = `geocode:calls:${orgId}:${todayKey()}`;
  const count = await redis.incr(key);
  if (count === 1) {
    await redis.expire(key, 60 * 60 * 26); // 26h, covers UTC-boundary drift
  }
  return count <= DAILY_CAP;
}

That code is correct. INCR is atomic, the TTL only gets set on the first increment of the day, and 26 hours safely outlives any clock skew across the day boundary. It had run unmodified for eight months.

The same Redis instance also holds the app's general object cache, geocode responses keyed by normalized address string, cached for a week so repeat lookups skip the provider entirely. Two weeks earlier, a bulk CSV import feature shipped, letting an org upload a spreadsheet of addresses and enrich all of them in one job. Nobody flagged it as a Redis change, because from the feature's point of view it was just calling the same cache-then-geocode path every other request used.

the scramble

First theory: the provider was down and lying about it. Their status page, a second team's independent monitor hitting the same endpoint, and a manual curl from a laptop off the office network all came back the same way, 429, with a response body naming our account's rate limit specifically. Not an outage. Rejected within six minutes.

Second theory: a compromised API key being hammered by something outside our own traffic. The provider's dashboard breaks call volume down by key. Nothing here was concentrated on one key, the load was spread across the account's entire shared quota, which looked like ordinary traffic that had simply gotten too loud in aggregate. Whatever was generating it lived inside our own system. Everyone's calls were hitting the account-wide ceiling, not one customer's.

the hunt

The internal rate limiter has its own dashboard, built from the same Redis counters. It showed one org, mid-CSV-import, at 340,614 geocode calls for the day against a cap of 5,000. The limiter had not blocked a single one of them.

Dead end: suspect a race in checkDailyCap itself, an off-by-one letting requests through around the boundary. A load test replaying the import's exact request pattern against a clean Redis instance held the cap at 5,000 every time, no drift, no double-counting. The code was fine in isolation.

The counter itself told a different story once someone checked it directly, mid-incident, on the actual production key:

> TTL geocode:calls:acme-corp:2026-09-27
(integer) -2
> GET geocode:calls:acme-corp:2026-09-27
(nil)

-2 means the key doesn't exist at all, hours into a day that had already generated hundreds of thousands of calls against it. The counter wasn't being read wrong. It kept disappearing.

INFO stats on the same instance had the next clue:

evicted_keys:38412
keyspace_hits:9821004
keyspace_misses:612330

Baseline for that stat, checked against the prior week's monitoring, was under 200 evictions an hour. CONFIG GET maxmemory-policy came back allkeys-lru, set months earlier specifically to stop the instance from OOM-crashing under normal cache growth instead of gracefully dropping cold entries. allkeys-lru evicts whatever key is least recently used to free memory, with no regard for whether that key carries a TTL, a business meaning, or anything else. A memory graph pulled from Datadog lined up used_memory crossing the configured ceiling at 10:47 UTC, sixteen minutes before the first 429, at the exact moment the CSV import started writing a week's worth of one-time cache entries for addresses it would likely never look up twice.

the find

allkeys-lru doesn't distinguish a cached geocode response from a billing-critical rate-limit counter. Both are just keys competing for the same memory budget. When the import's flood of cold cache writes pushed the instance over maxmemory, Redis's eviction sampling started reaping keys across the whole keyspace, including geocode:calls:acme-corp:2026-09-27, a key that still had roughly nineteen hours left on its TTL and had been written to seconds earlier by the exact job that needed it enforced.

Every eviction handed the import job a clean slate. The next call ran INCR against a key that no longer existed, got back 1, set a fresh 26-hour TTL, and the cap-checking logic saw a brand-new counter well under 5,000. The import's own retry loop, written to keep hammering the provider on any transient failure and never built to expect the internal limiter resetting mid-job, just kept going. Every reset bought it a fresh 5,000-call runway, over and over, for as long as memory pressure kept the eviction sampling landing on that key.

the fix

maxmemory-policy in Redis is set for the whole server, not per logical database and not per key prefix. Switching a SELECT'd database or namespacing keys more carefully wouldn't have changed which keys were eligible for eviction. The only real fix was moving the counters off the instance that needed permission to evict anything at all:

const limiterRedis = new Redis(process.env.LIMITER_REDIS_URL); // dedicated instance, noeviction

async function checkDailyCap(orgId) {
  const key = `geocode:calls:${orgId}:${todayKey()}`;
  const count = await limiterRedis.incr(key);
  if (count === 1) {
    await limiterRedis.expire(key, 60 * 60 * 26);
  }
  return count <= DAILY_CAP;
}

The new instance is small, sized generously against its actual working set, and configured with maxmemory-policy noeviction. If it ever approaches its ceiling, Redis starts rejecting writes with an out-of-memory error instead of silently deleting a counter, which fails loud and pages someone instead of failing invisible.

Second, independent of Redis entirely: the import job's retry loop now backs off on the provider's own 429 the same way it would need to for any other rate-limited dependency, instead of relying solely on the internal cap ever firing:

async function geocodeWithRetry(address, attempt = 0) {
  const res = await callProvider(address);
  if (res.status === 429 && attempt < MAX_RETRIES) {
    const delay = BASE_DELAY_MS * 2 ** attempt + jitter();
    await sleep(delay);
    return geocodeWithRetry(address, attempt + 1);
  }
  return res;
}

the aftermath

The counter did exactly what Redis promised: hold a value, expire it on schedule. Nobody had promised it would still be there before that.