How a Terminated Kubernetes Node Blackholed a Kafka Connection and Silently Failed a Third of Our Orders

03:20 UTC. PagerDuty pages the on-call engineer for checkout-api: 5xx rate on POST /orders just crossed 15%. Support hasn't reported anything yet. The dashboard has.

the setup

checkout-api is a FastAPI service that accepts an order, does a quick validation, and publishes it to order.submitted on Kafka for downstream fulfillment. It doesn't wait for a delivery confirmation before responding to the client, it hands the message to the producer's internal buffer and returns 202 immediately.

producer = Producer({
    'bootstrap.servers': 'kafka-headless.kafka.svc.cluster.local:9092',
    'queue.buffering.max.messages': 10000,
    'queue.buffering.max.ms': 50,
    'acks': 'all',
})

@app.post("/orders")
async def create_order(order: OrderIn):
    payload = order.model_dump_json().encode()
    try:
        producer.produce('order.submitted', key=str(order.id), value=payload)
    except BufferError:
        raise HTTPException(status_code=503, detail="order queue full, retry")
    return {"status": "accepted", "order_id": order.id}

Kafka runs as a three-broker StatefulSet in its own namespace, kafka-0, kafka-1, kafka-2, behind a headless service. Twenty-four partitions on order.submitted, eight led by each broker. The cluster had been stable for months, and node pool upgrades happened weekly without anyone thinking about them as a Kafka event at all.

the scramble

First theory: a bad deploy, checkout-api's deploy history showed nothing in the last six hours. Ruled out in under a minute.

Second theory: Postgres. checkout-api writes the order row before publishing to Kafka, so a database problem was a plausible first guess. Query latency, lock waits, connection pool saturation, all flat.

Third theory, and the one that ate the most time: a bad partition key. order.id is the Kafka key, and someone suggested a burst from one high-volume customer was hammering a single partition and backing up its broker. The failing requests, though, weren't clustered on any one customer, region, or order type. Whatever was failing was failing at random from the application's point of view, which usually means the cause isn't in the application at all.

The Kafka cluster dashboard itself looked fine the entire time: all three brokers up, zero under-replicated partitions, ISR full on every partition. That ruled out a cluster-wide problem and pointed the investigation back at the producer.

the hunt

checkout-api's logs were full of the same error, all from the last twelve minutes.

ERROR BufferError: Local: Queue full
  producer.produce('order.submitted', key='ord_9f21ac', ...) failed

queue.buffering.max.messages was set to 10,000, a deliberately conservative cap left over from an earlier incident where an unbounded producer buffer had contributed to an OOM kill. If the buffer was full, something downstream of the producer had stopped draining it. librdkafka exposes a stats callback with per-broker internals, and that's where the shape of the problem showed up.

"brokers": {
  "kafka-0.../0": { "state": "UP", "rtt": { "avg": 1421 }, "req_cnt": 88213 },
  "kafka-1.../1": { "state": "UP", "rtt": { "avg": 0 },    "req_cnt": 4102, "outbuf_msg_cnt": 9944 },
  "kafka-2.../2": { "state": "UP", "rtt": { "avg": 1358 }, "req_cnt": 90544 }
}

Broker 1's req_cnt hadn't moved across two consecutive polls of the stats callback, while its outbound queue climbed toward the 10,000 ceiling. Brokers 0 and 2 were sending and receiving fine. This wasn't a cluster problem or a hot-key problem, it was one specific broker connection that had stopped moving traffic while still reporting itself as UP.

A raw socket check on the checkout-api pod confirmed it.

ESTAB  0  45231  10.44.1.9:51882  10.42.3.17:9092
    cubic wscale:7,7 rto:912000 rtt:228000/8 mss:1420 retrans:0/47 unacked:31

10.42.3.17 was kafka-1's pod IP, or rather, it had been. The connection was still open from the kernel's point of view, forty-five kilobytes sitting unacknowledged in the send queue, retransmit timeout backed off to 912 seconds after 47 failed attempts. No RST, no FIN, ever. Just silence.

Kubernetes events for the node explained why.

03:06:58Z  Normal   NodeNotSchedulable  node/gke-prod-pool-7f3a2b  Node is now unschedulable
03:07:12Z  Normal   Killing             pod/kafka-1                Stopping container kafka
03:07:19Z  Warning  NodeNotReady        node/gke-prod-pool-7f3a2b  Node is not ready
03:08:03Z  Normal   Scheduled           pod/kafka-1                Assigned to gke-prod-pool-7f3a2b-m9qw
03:08:41Z  Normal   Started             pod/kafka-1                Started container kafka

A node pool upgrade drained kafka-1's node at 03:06:58 and the underlying VM was gone seven seconds after the container got its stop signal, before it finished closing its sockets. The pod rescheduled cleanly onto a new node with a new IP, and Kafka itself never noticed anything was wrong, because from Kafka's perspective nothing was: the new broker pod came up healthy and rejoined the cluster in under a minute.

The problem was entirely on the producer side. checkout-api's open connection to 10.42.3.17 had no peer left to answer it. Packets to a dead node don't get refused, they go nowhere, and a TCP stack has no way to tell "nobody's listening" apart from "the network is briefly congested." It just keeps retransmitting with exponential backoff until it gives up.

the find

Root cause: the producer's connection to broker 1 survived the node that ran it. Because the node was terminated rather than the container shutting down cleanly, no FIN or RST ever reached checkout-api, so librdkafka had no signal to close the connection and open a new one. It kept queueing messages for broker 1's eight partitions against a socket that was never going to accept them again.

Left alone, the kernel would have caught this on its own. Linux's default tcp_retries2 is 15, and against the backoff schedule that connection was on, that typically works out to somewhere around fifteen to twenty minutes before the socket gives up with a timeout error. checkout-api's own buffer cap of 10,000 messages filled long before that, in about twelve minutes at its steady produce rate for broker 1's share of traffic, and that's what actually paged anyone: not the dead connection itself, but the buffer built to protect against a different failure mode entirely.

the fix

The immediate fix was a rolling restart of checkout-api. Fresh pods open fresh connections, resolve kafka-1's DNS name again, and get the current IP. Recovery was under a minute once that shipped.

The real fix was making sure a blackholed connection gets detected fast instead of waiting on a kernel default tuned for general-purpose traffic, not for a service where every producer needs to notice a dead broker within seconds, not minutes.

producer = Producer({
    'bootstrap.servers': 'kafka-headless.kafka.svc.cluster.local:9092',
    'queue.buffering.max.messages': 10000,
    'queue.buffering.max.ms': 50,
    'acks': 'all',
    'socket.keepalive.enable': True,
})

securityContext:
  sysctls:
    - name: net.ipv4.tcp_keepalive_time
      value: "30"
    - name: net.ipv4.tcp_keepalive_intvl
      value: "10"
    - name: net.ipv4.tcp_keepalive_probes
      value: "3"

With keepalive enabled and tuned down from the OS default of two hours, a dead broker connection now gets probed every ten seconds and torn down within about a minute of going silent, well inside the buffer's headroom at normal traffic. librdkafka reconnects automatically once the socket errors out, and the reconnect does a fresh DNS lookup against the broker's stable hostname.

On the Kafka side, the StatefulSet got a longer termination grace period and a preStop hook so a future node drain gives the broker time to close its client connections cleanly before the node underneath it disappears.

terminationGracePeriodSeconds: 60
containers:
  - name: kafka
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "kafka-server-stop.sh"]

And the stats callback that had already exposed the per-broker breakdown during the hunt got turned into an actual alert, instead of something only pulled up manually mid-incident: any broker reporting UP with no successful request in 60 seconds, while its siblings are healthy, pages on-call directly, before the producer's buffer has any reason to fill.

the aftermath

Nothing here crashed. The node upgrade succeeded, the new broker pod came up healthy, and the Kafka cluster reported green the entire time. The failure was a connection that looked alive to everything that could see it and wasn't, and a kernel retry budget built for general traffic, not for a system where a few minutes of silence on one broker translates directly into failed customer orders.

The graceful shutdown hook is what should stop this specific sequence from happening again. The keepalive tuning is what stops the next one, whatever disappears out from under a connection next time, from taking twelve minutes to notice.