How a Webhook Backlog Drained Our Lambda Concurrency and Broke Checkout

14:32 UTC. PagerDuty pages the on-call backend engineer: checkout-confirm error rate above 20% for two consecutive minutes. Nobody deployed anything today. Nobody touched checkout this week. The function that's failing hasn't been opened in a pull request since July.

the setup

checkout-confirm is a small Lambda function sitting behind an API Gateway HTTP API. It validates the cart, calls the payment processor, and writes the order. Normal traffic is 15 to 20 concurrent invocations at peak. It has never needed more than that, so nobody ever set a concurrency limit on it, reserved or otherwise. On-demand Lambda functions default to drawing from the account's shared regional pool, and that had never been a problem worth thinking about.

A separate function, carrier-webhook-ingest, receives delivery-status updates from a regional shipping carrier. Every scan event, out-for-delivery, delivered, exception, arrives as a webhook, and API Gateway invokes the Lambda synchronously, one event in, one Lambda execution out. It also has no reserved concurrency configured. Also never a problem, because the carrier sends maybe 40 events a minute under normal load.

the scramble

First theory: RDS. Checkout writes to the orders table, and a connection pool exhaustion incident six months ago left everyone reflexively checking database connections first. RDS Proxy's connection metrics were flat, well under any limit. Dead end.

Second theory: a bad deploy, maybe from a dependent service. The deploy log for the last 48 hours showed nothing touching checkout, payments, or the shared API Gateway stage. Dead end.

Third theory, and the one that actually pointed somewhere: API Gateway's access logs showed the requests reaching Lambda fine, integration latency under 40ms, but the Lambda invocation itself returning an error before doing any work.

14:31:58 (a3f8-...) Endpoint response code: 429
14:31:58 (a3f8-...) Lambda invoke failed: TooManyRequestsException
14:31:58 (a3f8-...) Reason: Rate Exceeded.

Not a timeout. Not a runtime error inside the function. Lambda itself was refusing to start an execution. That's a throttle, and checkout-confirm wasn't anywhere near the invocation rate that would trigger one by itself.

the hunt

TooManyRequestsException at the invoke layer means one of two things: a function-level concurrency limit, or the account's regional pool is exhausted. Checking the function directly ruled out the first.

$ aws lambda get-function-concurrency --function-name checkout-confirm
{
    "ReservedConcurrentExecutions": null
}

No reserved concurrency set, which meant it draws from the shared pool, which meant the pool itself was the suspect.

$ aws lambda get-account-settings --query 'AccountLimit'
{
    "TotalCodeSize": 80530636800,
    "ConcurrentExecutions": 1000,
    "UnreservedConcurrentExecutions": 1000
}

1,000 concurrent executions is the AWS default regional limit, and it had never been raised because nothing in this account had ever come close to it. Pulling the account-level ConcurrentExecutions CloudWatch metric for the incident window showed it pinned at 1,000 starting at roughly 14:29 UTC, three minutes before the first checkout alert fired.

Something else was eating the pool. Breaking the same metric down by function pointed straight at it.

$ for fn in carrier-webhook-ingest checkout-confirm order-search notify-worker; do
    echo "$fn:"
    aws cloudwatch get-metric-statistics \
      --namespace AWS/Lambda --metric-name ConcurrentExecutions \
      --dimensions Name=FunctionName,Value=$fn \
      --start-time 2026-08-28T14:29:00Z --end-time 2026-08-28T14:35:00Z \
      --period 60 --statistics Maximum --query 'Datapoints[].Maximum'
  done

carrier-webhook-ingest:
[12.0, 940.0, 968.0, 971.0, 340.0, 61.0]
checkout-confirm:
[18.0, 21.0, 4.0, 2.0, 15.0, 17.0]
order-search:
[6.0, 3.0, 1.0, 2.0, 5.0, 6.0]
notify-worker:
[9.0, 7.0, 2.0, 3.0, 8.0, 9.0]

carrier-webhook-ingest jumped from 12 concurrent executions to 940 in the span of a minute, and stayed above 900 for two more. Every other function in the account, including checkout, got squeezed into whatever the pool had left, which for a two-minute window was effectively nothing.

Why did webhook ingestion spike 78x in a minute? The carrier's own status page had the answer, posted twelve minutes after our incident started: a regional outage from 08:00 to 14:00 UTC had queued every delivery-status event on their side, and at 14:29 they replayed the entire backlog in one burst. Roughly six hours of events, about 52,000 of them, landed on our webhook endpoint in under three minutes.

the find

Root cause: Lambda's concurrency limit isn't per-function by default, it's a single shared pool for the whole account and region. carrier-webhook-ingest was configured to scale as fast as invocations arrived, with no ceiling, because nobody had ever needed one. When the carrier replayed six hours of backlog at once, the function scaled to match, consumed 97% of the account's 1,000-slot pool, and left every other function, including the one that takes payment, fighting over the remainder.

Nothing about this looked like an ingest failure from the ingest side. The function wasn't erroring, it was doing exactly what an unbounded Lambda is designed to do under burst load. The damage showed up entirely on a function that had nothing to do with the carrier, which is why the first two theories both pointed at checkout and payments instead of a mailbox nobody was watching.

the fix

The immediate fix was a concurrency ceiling on the greedy function, so a burst on its side can never again consume the whole pool.

resource "aws_lambda_function" "carrier_webhook_ingest" {
  function_name = "carrier-webhook-ingest"
  # ...

  reserved_concurrent_executions = 150
}

A cap alone just changes who gets throttled during a burst, from checkout to the webhook processor itself. That's an acceptable tradeoff, since the carrier retries failed webhook deliveries and checkout does not get a second chance from an abandoned cart. But we also reserved a floor for checkout, since reserving concurrency for one function subtracts it from the shared pool and guarantees that capacity can't be taken by anything else.

resource "aws_lambda_function" "checkout_confirm" {
  function_name = "checkout-confirm"
  # ...

  reserved_concurrent_executions = 100
}

The bigger fix was architectural. A synchronous API Gateway to Lambda proxy has no buffer, every inbound event becomes an execution attempt immediately, which is exactly what turns a backlog replay into a concurrency spike. We put an SQS queue between the webhook endpoint and the processing Lambda, so a burst lands in the queue instead of spawning thousands of simultaneous executions.

resource "aws_lambda_event_source_mapping" "webhook_queue_trigger" {
  event_source_arn = aws_sqs_queue.carrier_webhooks.arn
  function_name    = aws_lambda_function.carrier_webhook_ingest.arn
  batch_size       = 25
  scaling_config {
    maximum_concurrency = 100
  }
}

The API Gateway route in front of the queue now just calls sqs:SendMessage directly through a service integration, no Lambda in that path at all. Ingestion drains the queue at a rate the account's pool can absorb no matter how large the burst on the carrier's side is, and a slow drain during a replay costs us minutes of delivery-status latency instead of a checkout outage.

We also added an account-level guardrail so the next unreserved function that spikes gets caught before it reaches another one.

resource "aws_cloudwatch_metric_alarm" "account_concurrency_high" {
  alarm_name          = "lambda-account-concurrency-utilization-high"
  namespace           = "AWS/Lambda"
  metric_name         = "ConcurrentExecutions"
  statistic           = "Maximum"
  period              = 60
  evaluation_periods  = 2
  threshold           = 700  # 70% of the 1000 regional limit
  comparison_operator = "GreaterThanThreshold"
}

the aftermath

About half of the 214 failed checkouts came back and retried successfully once the throttling cleared. The rest we can only infer from the drop against the same hour the week before, which is where the revenue estimate comes from, not a hard count.

The carrier's status page now gets a webhook subscription of its own, so a queued backlog replay shows up as an internal alert before it shows up as 52,000 events at our door. The account concurrency alarm hasn't fired since, though it came close once, during a load test someone forgot was scoped against production.