How a Claude Support Agent's Retry-on-Timeout Rule Issued 41 Duplicate Refunds in 53 Minutes

16:42 UTC, Friday. Someone in the #finance-ops Slack channel posts a screenshot of the day's Stripe refund total with one line: "this is $6k over what support says they approved." Nothing paged. No 500s, no error budget burned, no dashboard turned red. Every request involved had returned a clean 200.

the setup

Support tickets involving refunds were handled by a Claude-based agent sitting in front of the support queue. For anything under a fixed dollar threshold, it didn't just draft a response, it had a tool it could call directly:

{
  "name": "issue_refund",
  "description": "Issue a refund for a support ticket. Use once per ticket.",
  "parameters": {
    "ticket_id": { "type": "string" },
    "amount_cents": { "type": "integer" },
    "reason": { "type": "string" }
  }
}

The tool executor, the layer between the model and the actual refunds service, wrapped every call with an 8-second timeout. If a tool didn't return in time, the executor gave up waiting, wrote a synthetic error into the conversation, and let the model decide what to do next. The system prompt had one relevant line, added months earlier after a run of tickets stalled on a flaky email-lookup tool: "If a tool call fails, you may retry it once before escalating to a human." Reasonable for a read-only lookup. Nobody had revisited it for a tool that moves money.

The refunds service itself called Stripe's refund API with no idempotency key. It had never needed one. Under normal load its p50 was 400ms and its p99 was under 2 seconds, comfortably inside the executor's 8-second budget, so the gap between "the call is slow" and "the call failed" had never mattered.

the scramble

First theory, from whoever picked up the Slack thread: a support agent had manually approved refunds twice through the admin panel, maybe two people working the same queue. That was easy to check and wrong within a few minutes. The admin panel's audit log showed zero manual refund approvals that day. Every refund in the affected window had a source: agent tag.

Second theory: Stripe had a bug, or a webhook was firing twice and something downstream was treating each delivery as a new instruction. Also wrong. Stripe's dashboard showed two distinct re_ refund objects per ticket, not one refund with a duplicate webhook delivery. Two real refunds had been created, deliberately, by two separate API calls.

Third observation, around 17:05 UTC: every duplicated ticket had been opened in the same 53-minute window, and that window lined up with a shipping-carrier outage that had triggered a spike in delay-refund tickets. More tickets meant more concurrent issue_refund calls than the refunds service had ever seen at once.

the hunt

The orchestrator logs full tool-call traces per conversation. Pulling one affected ticket showed the shape immediately:

16:41:09.210  tool_call issue_refund(ticket=48213, amount=4200)
16:41:17.211  tool_executor: timeout after 8000ms, injecting error result
16:41:17.240  model: "The refund call timed out. Retrying once per policy."
16:41:17.244  tool_call issue_refund(ticket=48213, amount=4200)
16:41:18.501  tool_result: { status: "refunded", refund_id: "re_1P9x..." }
16:41:19.9xx  [async, arrives after executor gave up] refunds-service response
              for the FIRST call: { status: "refunded", refund_id: "re_1P8w..." }

The first call had not failed. It had taken 9.7 seconds under load, the refunds service was fine, just slower than usual because its connection pool to Stripe was being shared across far more concurrent requests than the pool was sized for. The executor's 8-second clock ran out first and reported a failure the model had no way to know was false. Claude followed its instructions exactly: one retry, no more. The second call landed on a now-recovering connection pool and returned in 1.3 seconds. By the time the first call's real response showed up in the trace, it looked like an afterthought. It wasn't. It had already reached Stripe and created a real refund object before anyone's clock said it hadn't.

await stripe.refunds.create({
  payment_intent: ticket.paymentIntentId,
  amount: amountCents,
  reason: 'requested_by_customer',
});

No idempotency key on that call meant Stripe had no way to know the second request was a retry of the first rather than a second, deliberate refund. It did exactly what it was told: created two.

the find

Root cause: the tool executor's timeout measured how long it was willing to wait, not whether the underlying call had succeeded. A timeout was surfaced to the model as an unqualified failure, and the model's retry-on-failure instruction, written for an idempotent read, was applied to a tool that creates a financial side effect with no deduplication underneath it. Three independently reasonable decisions, a bounded wait, a retry policy, a service with headroom, combined into a failure mode none of them would produce alone.

the fix

The first fix was at the only layer that could actually guarantee correctness: Stripe's own idempotency key, derived from something stable per logical refund rather than per HTTP attempt.

const idempotencyKey = `refund_${ticket.id}_${ticket.refundAttemptNonce}`;

await stripe.refunds.create(
  {
    payment_intent: ticket.paymentIntentId,
    amount: amountCents,
    reason: 'requested_by_customer',
  },
  { idempotencyKey }
);

refundAttemptNonce is written once per ticket the first time a refund is requested for it, not regenerated per call, so a timeout-triggered retry reuses the same key and Stripe returns the original refund object instead of creating a second one. That alone would have fully prevented this incident regardless of anything upstream.

Second, the tool contract changed so retries aren't the model's decision for mutating tools. The system prompt's retry line now only applies to tools explicitly marked idempotent in their schema; issue_refund isn't one of them. If the executor times out on a mutating call, it now polls the refunds service for the ticket's actual state before telling the model anything, instead of assuming silence means failure.

async def call_tool(tool, args, mutating):
    try:
        return await asyncio.wait_for(tool.invoke(args), timeout=TOOL_TIMEOUT_S)
    except asyncio.TimeoutError:
        if not mutating:
            return ToolResult(status="error", message="timed out")
        actual = await refunds_service.get_status(args["ticket_id"])
        if actual.refunded:
            return ToolResult(status="success", refund_id=actual.refund_id)
        return ToolResult(status="error", message="timed out, no refund created, safe to retry")

Third, the refunds service's Stripe connection pool was resized for the traffic spike class this incident revealed, and the executor's timeout was raised from 8 to 15 seconds to sit comfortably above the service's new, tested p99 under burst load rather than its old, quieter p99.

the aftermath

A reconciliation job now runs hourly, matching Stripe refund objects against ticket IDs and paging if any ticket has more than one. It would have caught this in under an hour instead of the 23 it actually took from first duplicate to the Slack thread that surfaced it, since nothing about two clean 200 responses looks wrong from inside either system.

The agent did exactly what it was told, twice. The instruction was the bug, not the model, and it had been sitting in the system prompt for months waiting for a tool important enough to make the gap matter.