How Kafka's Tombstone Retention Window Let a Revoked API Key Keep Working for 96 Hours
03:14 UTC. Security's on-call channel gets a message from the fraud-scanning job: a customer API key flagged and revoked four days earlier is still authorizing requests, all of them from ap-south-1. The revoke ticket has been closed as resolved since Monday.
the setup
authz-gateway sits in front of every API route and checks incoming keys against a
local cache before anything touches the database. It used to call Postgres on every request; that
was a flat 40ms added to p99 regardless of what the request actually needed. Six months earlier,
someone replaced it with a Kafka Streams state store hydrated from a compacted topic,
api-key-status, keyed by api_key_id. p99 for the auth check dropped to
4ms. An active key is a normal record. A revoked key is a tombstone, a record with the key present
and the value set to null, which is how compacted topics represent deletion.
function isKeyValid(apiKeyId) {
const record = keyStatusStore.get(apiKeyId); // local RocksDB state store
return record !== undefined && record.status === 'active';
}
The consumer group backing that store runs with group.instance.id set, static
membership, turned on months earlier after a rolling deploy triggered a full rebalance and briefly
spiked auth latency across every pod, not just the ones restarting. Static membership means a pod
that drops out during a deploy keeps its partition assignment reserved instead of forcing an
immediate reassignment. It was a good fix for that problem. It also meant nobody was watching for
what happens when a pod goes quiet for reasons that have nothing to do with a deploy.
the scramble
First move: check the source of truth. Postgres's api_keys table shows the key
correctly marked revoked_at four days ago. Whatever's wrong isn't the revoke itself,
it happened, it's recorded, it's just not everywhere it needs to be.
Second theory: the tombstone never made it onto the topic. A console consumer against
api-key-status from the revoke's approximate timestamp finds it immediately, a
null-value record for that exact key, committed and replicated. The event exists. The dead end
didn't cost much, five minutes to rule out, but it mattered: it meant the problem was downstream
of Kafka, not upstream of it.
Third theory, on-call's: a stale edge cache. This service doesn't have one on the authz path, ruled out by reading its own deployment topology rather than by testing anything.
the hunt
The fraud job's flagged requests all traced to three pods, all in ap-south-1, all part of the same
consumer group. kafka-consumer-groups.sh --describe against that group showed those
three partitions current, no lag, caught up to the log's high-water mark. Not stuck. Not behind.
Just wrong.
PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
4 88213 88213 0 authz-gateway-7f2a...
7 61940 61940 0 authz-gateway-7f2a...
9 73301 73301 0 authz-gateway-7f2a...Caught up with a wrong answer is a different bug than stuck behind. The next thing to check wasn't the topic, it was those three pods' history. Kubernetes events showed the answer: an ap-south-1-scoped NetworkPolicy change went out five days earlier as part of a node pool migration, and a rule ordering mistake in it blocked egress from those three pods to the Kafka broker subnet for roughly 30 hours before someone caught it in an unrelated ticket and reverted it. Nothing restarted those pods. Their liveness probe was an HTTP ping against the gateway's own port, which had nothing to do with whether it could reach Kafka, so it stayed green the entire time.
With no broker access, the consumer's background heartbeat thread stopped reaching the group
coordinator. Kafka correctly detected that after session.timeout.ms and reassigned
those three partitions to healthy members elsewhere, who kept the cache correct for everyone else.
The revoke tombstone was produced and consumed by those healthy members within seconds, exactly as
designed. It's what happened to the three isolated pods, once the network policy reverted and they
reconnected, that broke.
the find
Kafka Streams persists its state store to local disk and checkpoints the offset it last processed. On reconnect, if that checkpoint is still within the log, it doesn't replay from scratch, it does a delta restore: read forward from the checkpointed offset, apply whatever's new. That's the whole point of a local store, cheap recovery. The three pods' checkpoints were roughly 6 hours stale when the network came back, right around when the revoke had been written.
The topic's delete.retention.ms was left at Kafka's default, 86400000ms, 24 hours.
That setting exists specifically so a lagging consumer gets a grace window to see a tombstone
before the log cleaner physically removes it. The three pods were unreachable for roughly 30 hours,
past that window. By the time they reconnected and asked to resume from their checkpoint, the
tombstone that checkpoint needed to see had already been compacted away. The delta restore read
forward past where the tombstone used to be and found nothing for that key, so it never touched
the in-memory record still sitting there from before the isolation: active.
Nothing in that path throws an error. The consumer isn't lagging, the topic isn't corrupt, the restore completes successfully. It just completes with a state store that is now permanently wrong for one key, with no future event on the topic ever going to fix it, because as far as the topic is concerned that key's history now starts after the revoke.
the fix
Immediate: force the three pods to drop their local state directory and rebuild from the earliest offset rather than delta-restoring from a checkpoint that could no longer be trusted.
kubectl exec authz-gateway-7f2a-xyz -- rm -rf /data/kafka-streams/authz-gateway
kubectl delete pod authz-gateway-7f2a-xyzThen the two changes meant to keep this from being invisible again. First, a health check that actually reflects Kafka connectivity instead of just the local HTTP server:
let lastPollAt = Date.now();
consumer.on('poll', () => {
lastPollAt = Date.now();
});
app.get('/healthz', (req, res) => {
const staleMs = Date.now() - lastPollAt;
if (staleMs > 60_000) {
return res.status(503).json({ error: `kafka poll stale for ${staleMs}ms` });
}
res.status(200).json({ ok: true });
});
Second, delete.retention.ms on api-key-status went from 24 hours to 7
days, trading disk for a grace window wide enough to cover a multi-day network isolation, not just
a routine restart. It doesn't fix the underlying risk, a consumer down longer than the retention
window can still miss a tombstone, but it moves the failure mode from "a botched network policy"
to "something has been broken for a week and nobody noticed," which is a bar this team was willing
to accept.
the aftermath
- Static group membership solves rebalance storms during deploys, but it also means a consumer that's gone quiet for an unrelated reason keeps its assignment and its local state without raising anything from outside. It needs its own signal, not the assumption that a healthy pod implies a healthy consumer.
-
delete.retention.msis a grace window, not a guarantee. It's sized against how long a consumer might lag, not against how long one might be completely unreachable, and those are different failure modes with very different durations. - A local state store that restores by delta from a checkpoint is fast exactly because it trusts the checkpoint. That trust is only as good as the log's willingness to still contain everything between the checkpoint and now, and compaction doesn't make that promise indefinitely.
- A healthcheck that only proves a process is running, not that it's connected to the one dependency that determines whether its answers are correct, will report green through the exact failure it exists to catch.
Kafka never logged an error either. It deleted exactly what its retention setting told it to delete, on schedule, while the only three pods that needed to see it first were on the wrong side of a firewall rule.