How a Blocking Fraud-Check Call Triggered a Kafka Rebalance Storm and Duplicated 1,900 Orders

14:12 UTC. Consumer lag on order.created goes from flat to a wall in ninety seconds. PagerDuty pages the on-call engineer for inventory-service. By the time she has a terminal open, support is already reporting customers who got two order confirmation emails for one order.

the setup

Inventory-reservation-service consumes order.created off Kafka, runs each order through a third-party fraud-scoring API, and if it clears, reserves stock and publishes inventory.reserved. Three pods, one consumer group, twelve partitions. It had run this way for months without a single rebalance anyone noticed.

A deploy two days earlier added the fraud check itself, replacing a rule-based filter that had been flagging too many false positives. The call went straight into the message handler, on the same thread that calls poll().

while True:
    records = consumer.poll(timeout_ms=1000)
    for record in records:
        order = json.loads(record.value)
        score = fraud_client.check(order)   # synchronous HTTP call
        if score.passed:
            reserve_stock(order)
            publish_reserved_event(order)
    consumer.commit()

max.poll.interval.ms was set to 45 seconds, not the client default of five minutes. That tuning came out of an earlier incident where a stuck consumer sat silently for four minutes before anyone noticed, and shortening the interval was the agreed fix for catching that faster. The fraud API's own latency at the time was a steady 150 to 300 milliseconds, so a batch of 500 records finished in a few seconds and nobody connected the two settings.

At 14:10 UTC a flash sale drove order volume up roughly 8x. The fraud vendor's API, under its own load from every merchant running a sale that day, slowed from 200ms to 4 to 6 seconds per call. A batch of 500 records that used to take four seconds now took closer to forty minutes at that rate, and the loop never reached consumer.commit() before the poll thread went silent for longer than 45 seconds.

the scramble

First theory: the fraud vendor was down and the client was hanging on dead connections. Their status page showed green, and a manual curl against their endpoint from a laptop returned a response in under five seconds. Slow, not down.

on-call: fraud vendor status page all green, checked their /health directly,
          getting responses back just slow (~5s), not timing out

Second theory: a broker problem, maybe a partition leader flapping between brokers and forcing reconnects. Broker metrics showed no leader elections in the last hour and all partitions sitting in ISR with no under-replicated warnings.

Third theory, and the one that ate the most time: the duplicate emails were a bug in the email service itself, retrying sends on a timeout. The email pipeline's own logs showed no retries. It had been asked to send the same order confirmation twice, from two separate calls, several minutes apart. Whatever was duplicating work was upstream of email entirely.

the hunt

Consumer lag on order.created was climbing in a sawtooth, not a straight line: up for a couple of minutes, a brief drop, then up again. That shape usually means partitions are being reassigned repeatedly, not that one consumer fell behind and stayed behind. The broker logs confirmed it.

[GroupCoordinator] Preparing to rebalance group inventory-reservation-service
  reason: Removing member consumer-2-a91f (session timed out)
[GroupCoordinator] Stabilized group inventory-reservation-service generation 47
[GroupCoordinator] Preparing to rebalance group inventory-reservation-service
  reason: Removing member consumer-1-c02e (session timed out)
[GroupCoordinator] Stabilized group inventory-reservation-service generation 48
[GroupCoordinator] Preparing to rebalance group inventory-reservation-service
  reason: Removing member consumer-3-f61b (session timed out)

Six rebalances in twelve minutes, one per pod, in rotation. The consumer-side logs explained why each one happened.

WARN [Consumer clientId=inventory-2] This member will leave the group
     because consumer poll timeout has expired. This means the time
     between subsequent calls to poll() was longer than the configured
     max.poll.interval.ms, which typically implies that the poll loop is
     spending too much time processing messages.

Each pod was blocked inside the fraud-check call for a batch of records, missed its 45-second poll deadline, and got fenced out of the group. Its partitions reassigned to whichever pod was free, which immediately started pulling its own batch of 500 records, hit the same slow API, and got fenced in turn a few dozen seconds later. Three pods took turns being evicted, none of them ever finishing a batch cleanly enough to commit past where the previous owner had already gotten to.

the find

Root cause: a synchronous, unbounded HTTP call inside the Kafka poll loop, combined with a max.poll.interval.ms that had been tuned down for a different failure mode. The setting that was supposed to catch a stuck consumer faster instead turned a slow dependency into a guaranteed eviction, and eviction mid-batch meant partitions kept changing hands before any consumer could commit offsets past the point where it had already produced side effects.

Every record fully processed before an eviction, fraud check passed, stock reserved, email sent, but not yet committed, got replayed by whichever pod picked up that partition next. The fraud check and stock reservation had no dedupe key, so replays produced a second reservation and a second confirmation email for the same order. Some orders got processed three times across the six rebalances before the storm settled.

the fix

The immediate fix was to get the fraud check off the poll thread entirely, so a slow dependency can never cost the consumer its group membership.

executor = ThreadPoolExecutor(max_workers=8)

while True:
    records = consumer.poll(timeout_ms=1000)
    if records:
        futures = [executor.submit(process_order, r) for r in records]
        for f in futures:
            f.result(timeout=8)   # bounded, independent of poll interval
    consumer.commit()

def process_order(record):
    order = json.loads(record.value)
    score = fraud_client.check(order, timeout=3)
    if score.passed and not already_reserved(order.id):
        reserve_stock(order)
        publish_reserved_event(order)

already_reserved(order.id) checks a Postgres table with a unique constraint on order_id, written in the same transaction as the reservation. That closes the actual hole: even if a rebalance replays a record, the reservation and the email only fire once.

The fraud client also got a hard 3-second timeout and a circuit breaker: past a threshold of consecutive slow responses, orders route to a manual review queue instead of blocking on the vendor at all. And max.poll.interval.ms went back up to two minutes, with a separate, purpose-built alert on the exact log line that started this incident, consumer poll timeout has expired, instead of relying on a short interval to surface a stuck consumer.

the aftermath

Nothing here was a crash. Every pod was healthy, the broker was healthy, and the fraud vendor was technically up the whole time. The failure was purely about time budgets: a poll loop that does unbounded synchronous work is one slow dependency away from evicting itself, and an eviction mid-batch is a replay waiting to produce duplicate side effects if nothing downstream is idempotent.

The dedupe check would have contained this to a lag spike with no customer impact. The timeout and circuit breaker on the fraud client are what stop the next slow dependency from turning into another rebalance storm at all.