How a Middleware Reorder Broke Every Stripe Webhook Signature and Left $38,400 Stuck Pending

14:07 UTC. A deploy ships a small request-logging middleware, nothing dramatic, the kind of change that doesn't even get its own paragraph in the PR description. By 14:48 UTC, support has three tickets from customers who paid and are still staring at "Order Pending." Stripe's dashboard shows the charges as succeeded. Our database disagrees.

the setup

Orders get marked paid by a webhook, not by the checkout request itself. The client calls Stripe directly to confirm the payment intent, and separately, Stripe calls our /webhooks/stripe endpoint with a payment_intent.succeeded event. That webhook is the only thing that flips an order from pending to paid. It's a deliberate design, trusting the client's redirect to mean "paid" is how you end up shipping orders nobody paid for.

Verifying that webhook is genuinely from Stripe means recomputing an HMAC-SHA256 signature over the exact bytes of the request body and comparing it to the Stripe-Signature header. Stripe's SDK handles the comparison, but it needs the raw, unparsed body, not req.body after Express has already turned it into an object. Re-serializing JSON changes whitespace and key order, and a signature computed over the wrong bytes will never match, no matter how correct the payload looks.

That's why the webhook route had its own middleware, scoped only to that path: express.raw({ type: 'application/json' }), which hands the handler a Buffer instead of a parsed object. It had worked, untouched, for over a year.

the scramble

First theory: Stripe was having a bad day. status.stripe.com was fully green, and the Stripe dashboard showed the events as delivered, with a 400 response from our side. Stripe was sending them fine. We were the ones rejecting them.

Second theory: somebody had rotated the webhook signing secret and forgotten to update it in the environment. That felt promising for about four minutes, until the value in the secrets manager was diffed against the one in the Stripe dashboard. Identical. Nobody had touched it in months.

The actual clue was sitting in the application logs the whole time, buried under a normal volume of noise until someone grepped for the webhook route specifically.

StripeSignatureVerificationError: No webhook payload was provided.
  Did you forget to use body-parser's raw() middleware or otherwise
  read the request body into a Buffer before verification?

Stripe's SDK throws that exact message when it gets something other than a raw buffer. The raw body middleware was still there, still scoped correctly to the route. So why wasn't it seeing a buffer anymore?

the hunt

The webhook route code hadn't changed in the deploy. git log -p on the route file came back empty for the last two weeks. So the bug wasn't in the route, it was in something that ran before the route.

That request-logging middleware, the one nobody thought twice about, needed req.body populated to log request payloads for debugging. To get that, the PR added a global body parser ahead of the route registrations:

app.use(express.json());
app.use(requestLoggingMiddleware);

// ...registered further down, unchanged...
app.use('/webhooks/stripe', express.raw({ type: 'application/json' }), stripeWebhookRouter);

Express middleware runs in registration order, for every path, unless it's explicitly scoped. express.json() at the top has no path filter, so it runs on every incoming request including /webhooks/stripe, reads the request stream, and parses it into an object. A Node.js request stream can only be consumed once. By the time execution reached the route's own express.raw() middleware, the stream was already drained. It handed the handler an empty buffer, and Stripe's SDK correctly refused to verify a signature against nothing.

This wasn't intermittent and it wasn't a subset of events. Every single Stripe webhook, of every type, had been failing since the deploy went out at 14:07. It took 41 minutes to notice because there was no alert on the webhook endpoint's error rate, only on the overall API's, and one route returning 400s didn't move that needle.

the find

Root cause: a global, unscoped body parser registered ahead of a route that depended on reading the raw request stream itself. Nothing about the new middleware was wrong in isolation, logging request bodies is an ordinary thing to want. The mistake was registering it globally instead of scoping it to the routes that actually needed a parsed body, which silently broke the one route built around a different assumption.

Stripe retries failed webhook deliveries on a backoff schedule, so no event was permanently lost, but every retry during the outage window hit the same broken middleware order and failed the same way. Orders stayed pending until each event's retry eventually succeeded, hours later, or until we intervened directly.

the fix

The immediate fix was reordering the middleware so the webhook route's raw parser runs before anything global touches the body, and excluding that path from the global JSON parser entirely.

// Raw body capture for the webhook route, registered first, so nothing
// downstream can consume the stream before Stripe's signature check does.
app.use('/webhooks/stripe', express.raw({ type: 'application/json' }), stripeWebhookRouter);

// Global JSON parsing for everything else.
app.use(express.json());
app.use(requestLoggingMiddleware);

We also added a verify callback to the global parser, so any future route that needs the raw bytes for something like signature checking has them available on req.rawBody without depending on middleware order at all:

app.use(express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf;
  },
}));

For the backfill, we pulled every payment_intent.succeeded event from Stripe's Events API for the outage window and replayed them through the internal handler directly, rather than waiting on Stripe's retry schedule to catch up on its own. While writing that replay script, I needed to sanity-check a signature by hand against a couple of edge-case payloads before trusting the automated version, and used nodique.com's webhook signature verifier to do it rather than spinning up a throwaway Node script for a five-minute check. It's in the incident runbook now as the fast path for exactly that kind of spot check.

the aftermath

The webhook route's raw parser now runs before anything else touches the request, and the canary has caught this exact failure mode twice in staging since, both times from unrelated middleware additions that would have shipped the same bug again.