How One request.body() Call in Middleware Silently Zeroed 874 SKUs in 41 Minutes

15:04 UTC. The page isn't an error rate, every route is still returning 200. It's a business metric: available_sku_count down 38% in six minutes, no promo running, no deploy freeze broken, nothing in the error budget dashboard even blinking.

the setup

/webhooks/inventory-sync is how the warehouse partner tells us what's actually on the shelf. Four times a day it POSTs a batch of SKU quantities for one warehouse, and the batch is treated as authoritative for that warehouse: any SKU not present in the payload gets set to zero, on purpose, because "not listed" means "this warehouse doesn't have it anymore." That's the correct behavior for a real batch. It's catastrophic for an empty one.

@app.post("/webhooks/inventory-sync")
async def sync_inventory(request: Request):
    body = await request.body()
    payload = json.loads(body) if body else {}
    quantities = payload.get("skus", {})
    await inventory_service.replace_batch(
        warehouse_id=payload.get("warehouse_id"),
        quantities=quantities,
    )
    return {"status": "ok"}

Two weeks earlier, a different team shipped /webhooks/returns, a new endpoint for the same 3PL partner to report merchandise returns. It needed signature verification, since return events carry a refund amount and the partner signs every payload with an HMAC secret. The middleware that checks it was registered globally, because add_middleware doesn't have a simple per-route scope in Starlette, and nobody thought that mattered yet:

class HMACAuthMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        body = await request.body()
        if request.url.path == "/webhooks/returns":
            expected = hmac.new(SECRET, body, hashlib.sha256).hexdigest()
            if expected != request.headers.get("x-signature"):
                return Response(status_code=401)
        return await call_next(request)

app.add_middleware(HMACAuthMiddleware), one line, no path filter at registration. The path check inside dispatch runs after the body is already read.

the scramble

First theory: the 3PL sent a bad file. Their delivery portal logs every batch with a byte count, and the batch at 15:00 UTC was 14,832 bytes, in line with every prior sync that day. Whatever went wrong, it wasn't a bad file on their end.

Second theory: a stale read cache showing zeros that weren't real. The inventory read path does cache aggressively, five-minute TTL on the storefront-facing counts. But the cache was doing its job correctly, serving exactly what was in Postgres. The zeros were real. Something had actually told the database to zero those rows.

the hunt

sync_inventory already logged a line on every call, added months back for a different investigation and never removed:

14:47:12 INFO sync_inventory warehouse_id=WH-14 skus_received=1897
15:00:03 INFO sync_inventory warehouse_id=WH-14 skus_received=0

1,897 SKUs at 14:47, zero at 15:00, same warehouse, same partner, same endpoint. The load balancer's access log for the 15:00 request showed content-length: 14832, the same 14,832 bytes the partner's own delivery log recorded. The bytes arrived over the wire intact. They just weren't there anymore by the time the handler tried to read them.

Deploy history for that morning showed one change touching anything shared: the HMACAuthMiddleware for /webhooks/returns, merged and deployed at 14:32 UTC. Disabling it on a single canary pod and re-sending the same 15:00 batch by hand confirmed it in under a minute, skus_received came back as 1,897 with the middleware off, zero with it on.

the find

Starlette's BaseHTTPMiddleware runs every downstream handler through a second task with its own copy of the request's receive channel, so it can stream the response back through call_next. If the middleware itself calls await request.body() before deciding whether it even cares about this request, that call drains the underlying ASGI stream directly. The request object the route handler eventually sees shares the same scope, but there's no body left in the channel to hand it, and nothing that replays what was already consumed.

The path check inside dispatch was correct. It just ran one line too late. await request.body() executed for every request the middleware saw, which was every request in the app, before the function ever checked whether the path was /webhooks/returns. /webhooks/inventory-sync never triggered the signature check, and never needed to. It still paid the cost of the read.

And the inventory handler's own defensiveness turned a drained stream into a silent disaster instead of a loud one: json.loads(body) if body else {} was written to tolerate a genuinely empty batch as a no-op. An empty batch from the partner has never happened and was never supposed to mean anything. A batch that arrived full and got read as empty meant "zero out this warehouse," and the handler had no way to tell the two apart.

the fix

Three changes, none of them clever. First, the path filter moves ahead of the body read, so requests the middleware doesn't care about never pay for it:

class HMACAuthMiddleware(BaseHTTPMiddleware):
    SIGNED_PATHS = {"/webhooks/returns"}

    async def dispatch(self, request: Request, call_next):
        if request.url.path not in self.SIGNED_PATHS:
            return await call_next(request)

        body = await request.body()
        expected = hmac.new(SECRET, body, hashlib.sha256).hexdigest()
        if expected != request.headers.get("x-signature"):
            return Response(status_code=401)

        request._body = body  # cache so the downstream handler can still read it
        return await call_next(request)

Second, for the one path that actually does need to read the body in middleware, the read gets cached back onto the request object explicitly, the line above, rather than assuming a downstream request.body() call will find anything left to read.

Third, and the one that would have kept this from becoming an incident even with the middleware bug still live: sync_inventory stopped treating an unparseable or empty body as an empty, valid batch.

@app.post("/webhooks/inventory-sync")
async def sync_inventory(request: Request):
    body = await request.body()
    if not body:
        raise HTTPException(status_code=400, detail="empty inventory sync payload")

    payload = json.loads(body)
    quantities = payload.get("skus")
    if not quantities:
        raise HTTPException(status_code=400, detail="sync payload had no SKUs")

    await inventory_service.replace_batch(
        warehouse_id=payload["warehouse_id"],
        quantities=quantities,
    )
    return {"status": "ok"}

A full-replace batch API is only safe if "nothing here" is treated as suspicious, never as routine. An audit of every other global middleware in the app turned up two more call sites reading request.body() before checking whether the request in front of them was one they were meant to touch. Both now filter first.

the aftermath

Every request that hit the bug returned 200. The bug was never in what the app said back. It was in what the app quietly decided not to read.