How prefetch_count: 0 on One Queue Blocked Every Publisher on the Broker

09:14 UTC. PagerDuty: "checkout-api: publish to order-events timing out, 40% error rate." Two minutes later, a second page: "inventory-sync: AMQP publish timeout." Then a third, for a service that hadn't touched RabbitMQ config in eight months. Three unrelated services, same broker, same moment.

the scramble

First theory: the broker was down. It wasn't. rabbitmqctl status from a bastion host came back clean, node running, cluster of three all green. Whatever this was, it wasn't a crash.

Second theory: a network problem between the app tier and the broker's load balancer. On-call pulled the target group health checks. All three broker nodes were healthy, TCP connections were establishing fine, TLS handshakes completing in under 50ms. Publishers were connecting. They just weren't able to publish. Dead end, but a useful one, it narrowed the problem to something the broker itself was doing on purpose.

Third theory came from the RabbitMQ management UI, which nobody had opened yet because the first instinct was always to check the app side first.

Alarm: memory alarm on rabbit@broker-node-2. Publishers throttled.

RabbitMQ has a memory high watermark, 0.4 of available RAM by default on most managed deployments, and when a node crosses it, the broker doesn't crash or drop messages. It blocks every connection trying to publish, cluster-wide, until memory drops back under the threshold. That's the mechanism working as designed. The question was what had filled memory on a broker that had been running unremarkably for months.

the hunt

The Overview tab's per-queue breakdown answered that in about ten seconds.

Queue                    Ready    Unacked    Total
shipping-labels          1,204    812,447    813,651
order-events                 3          0        3
inventory-sync-events        0          2        2
notifications                0          0        0

shipping-labels had over 800,000 unacknowledged messages. Unacked messages stay fully resident in the broker's memory until the consumer acks or nacks them, they aren't written to disk the way ready messages on a lazy queue are. Eight hundred thousand of them, most carrying a full order payload, was more than enough to push node-2 over its watermark on its own.

The consumer for that queue was a Python worker using pika, one process, one channel, calling a carrier's label-generation API for every message.

channel.basic_qos(prefetch_count=0)  # "no limit, let it fly"

def on_message(ch, method, properties, body):
    order = json.loads(body)
    label = carrier_client.create_label(order)  # no timeout kwarg
    save_label(order["order_id"], label)
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(queue="shipping-labels", on_message_callback=on_message)

prefetch_count=0 doesn't mean "a sane default." In RabbitMQ's AMQP implementation it means unlimited, the broker will hand the consumer as many unacked messages as it can pull, with nothing capping how far in-flight work can get ahead of actual processing. Whoever wrote this worker had picked 0 assuming it meant something like "no artificial limit, let the client library figure out a reasonable batch size." It doesn't. It means no limit at all.

That had been survivable for months because the carrier's API usually answered in 200 to 400ms, fast enough that the ack rate kept pace with the publish rate and unacked count stayed in the low hundreds. This particular morning, the carrier had a degraded API window of their own, not a full outage, just p50 latency climbing past four seconds and p99 past twenty. The create_label call had no timeout kwarg, so a slow request didn't fail fast, it just sat there. One consumer, one channel, meant every message was processed serially. At twenty seconds a message with no upper bound on how many could queue up unacked behind it, the backlog didn't grow, it exploded.

CloudWatch logs for carrier_client confirmed the timing. The carrier's own status page later posted a retroactive incident for the same window, a caching layer they'd deployed that morning had a cold-start problem. Nobody on our side would have known that at the time, and it didn't matter for the fix, it was the trigger, not the root cause.

the find

Root cause: an unbounded prefetch count on a single consumer let unacked messages accumulate without any backpressure, and because unacked messages are memory-resident, that accumulation was directly convertible into broker RAM pressure. When a slow, unrelated downstream dependency turned processing latency for that one queue from milliseconds into seconds, there was nothing stopping the backlog from growing until it tripped the broker's memory alarm, at which point RabbitMQ's flow control protected the broker by blocking publishes cluster-wide. A local slowdown in one worker became a global outage for every service that talked to that broker, because memory is a shared resource across every queue on the node, and the alarm has no concept of which queue caused the problem.

the fix

First, an actual prefetch limit, sized to what one consumer process can reasonably hold in-flight without becoming a de facto unbounded buffer:

channel.basic_qos(prefetch_count=20)

def on_message(ch, method, properties, body):
    order = json.loads(body)
    try:
        label = carrier_client.create_label(order, timeout=5.0)
    except CarrierTimeoutError:
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=True)
        return
    save_label(order["order_id"], label)
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(queue="shipping-labels", on_message_callback=on_message)

With prefetch at 20, the broker will never hand this consumer more than 20 unacked messages regardless of how slow processing gets. Ready messages pile up in the queue instead, which is fine, RabbitMQ writes those to disk once a queue crosses its lazy-queue threshold, so a growing backlog of ready-but-not-yet-delivered work costs disk, not RAM.

Second, an explicit timeout on the carrier call, with a nack-and-requeue on failure instead of letting a hung request hold a delivery tag indefinitely.

Third, we scaled the consumer count for that queue from one process to four, so a temporarily slow downstream degrades throughput proportionally instead of serializing every order behind a single connection.

Fourth, an alert on the metric that would have caught this before the memory alarm did:

queue: rabbitmq.queue.messages_unacknowledged
scope: queue:shipping-labels
threshold: > 500 for 5 minutes
message: "shipping-labels unacked backlog growing. Check downstream carrier latency before
  this trips a broker memory alarm and blocks every publisher on the cluster."

We also added the broker's own memory alarm metric to the same dashboard, but the queue-level unacked alert is the one meant to fire first, days or hours before any single queue's backlog gets anywhere near broker-wide impact.

the aftermath

Nothing was lost. Every blocked publish either succeeded on retry once the alarm cleared or failed loudly enough to be visible. That's the point of flow control: it protects the broker at the cost of availability rather than silently dropping data. But availability was the thing that paged three on-call engineers for a problem that started in a single worker's queue configuration.

The carrier's caching layer stabilized on its own later that morning, unrelated to anything we changed. Our fix didn't touch their API. It just made sure the next time any downstream call gets slow, whichever queue notices first can absorb it without taking the broker, and everyone else publishing to it, down too.