How a 64-Bit Order ID Parsed by JSON.parse Sent 11 Refunds to the Wrong Orders
10:07 UTC. Finance posts in #payments-ops: the daily Stripe refund total is $1,842 higher than the refund total in our own orders table. Two minutes later a support lead adds a second problem to the thread. "The Refund button in the admin tool says order not found about half the time. Searching for the order by number and refunding from there works."
the scramble
Two people get pulled in. The finance mismatch is the louder problem because it means money moved and our books do not say where. The 'order not found' toast is the cheaper one to look at first, so that is where I start.
First theory: replica lag. The admin tool reads from a replica, the order was created minutes ago, and the refund endpoint hits the primary. If the replica had not caught up, the list would not show the order at all, and it does. Replica lag is also flat at under 200 ms. Ruled out.
Second theory: the deploy from three days ago broke the refund route. Three days ago we shipped a new order ID scheme, so it is the obvious suspect. But the route works when support reaches it through the search box, and the same route fails from the list. Same code, same endpoint, different result depending on how you got there.
Third theory: a double click. The button has no debounce and a duplicate refund could explain extra Stripe money. It cannot explain the 404, and the duplicate-refund guard (an idempotency key on the order) would have rejected a second call anyway.
the hunt
I stop theorizing and look at one failing request. Browser devtools, admin tool, click Refund on an order that fails. The request line is:
The order I clicked, according to the page, is number 14663680000001737. The list
response, in the network tab, also says 14663680000001737. The request path says
...736. The browser changed the number between receiving it and sending it.
My first reaction is that I am misreading the digits, so I check a second order. List says
14663680000001739, refund request goes to ...740. A third: list says
...738, request says ...738, and that one succeeds. Odd IDs change, even
IDs do not.
Nothing in the admin tool's own code touches the ID. It is read from the response and interpolated into a URL:
Three lines, no arithmetic, and the number still changes. So it has to be the parse. I paste the raw bytes of the list response into a Node REPL.
the find
A JavaScript number is a 64-bit float with a 53-bit mantissa. Every integer up to
2^53 - 1 is exact. Above 2^53, which is 9,007,199,254,740,992, the float
can only represent every second integer. Above 2^54 it is every fourth, and so on. Our
new IDs are around 1.47e16, which sits between 2^53 and 2^54, so every odd
ID has no exact representation and gets rounded to an even neighbor.
The ID change three days ago moved orders from a serial integer to a time-ordered ID, built like this:
Postgres stores it as bigint, FastAPI serializes it as a JSON integer, Python never
complains because Python integers are arbitrary precision. The only component that cannot hold the
value is the browser, and it fails without an error.
That explains the 404s: roughly half of orders have an odd ID, and the rounded value usually does not exist. It does not yet explain the finance number, because a 404 refunds nothing. For that I need the case where the rounded ID does exist.
It exists when two orders were created in the same millisecond on the same shard, which gives
consecutive IDs. Our wholesale partner API inserts a batch of up to 300 orders in one transaction,
so those orders get IDs that are adjacent integers. Order ...737 rounds down to
...736, which is a real order, from the same partner, and the refund endpoint
refunds it. I query the refund table against the partner orders from the last three days:
Eleven rows, totaling $1,842. Refunds on even-numbered partner orders with no support ticket against them. Each one was meant for the odd order right next to it.
the fix
The right fix is on the API side. A 64-bit identifier should not be a JSON number at all, because we cannot control every client that parses it. We serialize order IDs as strings and keep them as integers everywhere else:
Path parameters were already fine since they arrive as text, and FastAPI parses them into
int on the server. On the admin tool the type changes to id: string, and
the compiler then points at every place that did arithmetic or comparisons on it.
To stop this class of bug from coming back I added a contract test that walks every JSON response in our API test suite and fails if any integer is outside the safe range:
It ran against the existing suite right away and found two more endpoints leaking the new ID as a number: the webhook payload and the CSV export's JSON mode. Both were fixed in the same change.
the aftermath
- Impact over three days: about 2,300 failed refund clicks (support worked around them with the search box), and 11 refunds applied to the wrong partner orders, $1,842 in total.
- The 11 refunds were reversed by hand. Seven of the affected customers had already received a "your refund is on its way" email for an order they had not asked to refund, and got an apology.
- The string-ID fix shipped at 13:40 UTC. The contract test now runs in CI on every API change.
- The refund endpoint now requires the amount in the request and rejects it if it does not match the order, so a mistargeted request fails loudly instead of succeeding on the wrong row.
- Runbook line: if a number in the browser differs from the number in the response bytes, suspect
JSON.parseand anything above 9,007,199,254,740,991.
There was no bug in the refund code. The API sent a valid integer and the browser parsed it the way the spec says to. For three days the only symptom anyone noticed was a button that failed half the time. The failures were the harmless ones. The clicks that succeeded did the damage.