How Reusing a Protobuf Field Number During a Rolling Deploy Auto-Approved High-Risk Orders

10:21 UTC. Someone on the fraud team notices the manual review queue has gone quiet. Not slow, quiet. Zero items in seven minutes, on a Tuesday morning when the queue usually runs a steady trickle of twenty to thirty orders an hour. Nobody paged for it, because an empty queue doesn't look like an outage. It looks like a good morning.

the scramble

First theory: the fraud-scoring model got faster, or got better, and fewer orders are tripping the threshold that sends them to manual review. Someone pulls up the model-serving dashboard. Latency's flat, request volume's flat, score distribution for the last hour looks the same shape as yesterday's. Nothing changed there.

Second theory: a feature flag. The team has a flag, skip_manual_review_legacy_partners, that exempts a small, specific list of trusted wholesale accounts from the fraud queue entirely. If that flag's rollout percentage got bumped by accident, this is exactly what it would look like. The flag service's audit log for the last six hours shows no changes. Still at 2%, still pointed at the same eleven account IDs it's always pointed at.

Third theory, and the one that stalls the room for a few minutes: maybe nothing's wrong, and order volume is just genuinely low this morning. Someone checks total order count. 1,840 orders in the last hour, completely normal for a Tuesday. Orders are flowing. They're just not landing in the queue that's supposed to catch the risky ones.

the hunt

With the model and the flag both clean, the hunt moves to the orders that should have been flagged and weren't. Someone writes a quick query against the orders service's own risk-tier field, which is set independently of the fraud queue and should always agree with it.

SELECT risk_tier, COUNT(*), SUM(CASE WHEN sent_to_manual_review THEN 1 ELSE 0 END)
FROM orders
WHERE created_at > now() - interval '60 minutes'
GROUP BY risk_tier;

 risk_tier | count | sent_to_review
 LOW       |  1,612 |  0
 MEDIUM    |    148 |  0
 HIGH      |     68 |  0
 CRITICAL  |     12 |  0

Every single MEDIUM, HIGH, and CRITICAL order in the last hour skipped review. Not some of them, all 228 of them. That's not a scoring drift, that's a switch flipped to off for an entire category. But the flag is clean and the model is clean, so something between the two services is lying.

Orders and fraud-review talk over gRPC, one call per order, carrying a RiskTier enum set by the orders service. Someone checks whether this started at a specific time rather than drifting. It did: 10:14 UTC, almost to the minute. And 10:14 UTC is also when the orders service's rolling deploy for an unrelated change, a shipping-cost rounding fix, started rolling pods.

That's the first real lead, but it still doesn't explain the mechanism. The shipping fix touched nothing related to fraud. Someone diffs the two proto definitions, the one baked into the old pods still running and the one in the new build going out, looking for anything that moved.

the find

The orders.proto file has a field that was deprecated eight months ago:

message OrderEvent {
  string order_id = 1;
  string account_id = 2;
  // ...fields 3-6 omitted...

  // DEPRECATED 2026-02-11, legacy wholesale fee program ended.
  // Do not use. Will be removed.
  bool legacy_partner_waiver = 7;
}

The comment says "do not use." The comment is not protobuf. Nothing in the schema itself stops a field number from being reused, because a comment isn't a compiler constraint. Three weeks ago, a different engineer on a different team added fraud tiering to the same message, picked the next available number by habit, and field 7 looked available. It was not marked reserved, so nothing flagged it.

enum RiskTier {
  LOW = 0;
  MEDIUM = 1;
  HIGH = 2;
  CRITICAL = 3;
}

message OrderEvent {
  string order_id = 1;
  string account_id = 2;
  // ...fields 3-6 omitted...

  RiskTier risk_tier = 7; // reused field number 7
}

Protobuf's wire format encodes both a bool and an enum as wire type 0, a plain varint. A decoder doesn't know or care which Python/Go struct field that varint is destined for, only that field number 7 is a varint and here's its value. Someone pulls the raw bytes off the wire during the deploy window to confirm it directly.

$ protoc --decode_raw < order_event_capture.bin
1: "ord_8f3ac1"
2: "acct_44210"
7: 2

# New pod encodes field 7 as RiskTier.HIGH (value 2).
# Old pod, still running the pre-deploy binary, decodes field 7
# as legacy_partner_waiver. Any nonzero varint reads as boolean true.

During the rolling deploy, new pods were producing OrderEvent messages with risk_tier encoded at field 7. Old fraud-review pods, not yet rolled, were still running the binary that interpreted field 7 as legacy_partner_waiver. Any order tagged MEDIUM, HIGH, or CRITICAL (values 1, 2, or 3, all nonzero) decoded as legacy_partner_waiver = true on those old pods, which triggers an auto-approve path that was built for trusted wholesale accounts and was never supposed to see a risk tier at all. The higher the actual fraud risk, the more certain the auto-approve.

the fix

The immediate fix was finishing the rolling deploy, which naturally stopped new waivers the moment every fraud-review pod ran the new binary and started treating field 7 as the enum it was supposed to be. That took four more minutes. The real fix had three parts.

First, remove the dangerous field properly instead of commenting around it:

message OrderEvent {
  string order_id = 1;
  string account_id = 2;
  // ...fields 3-6 omitted...

  reserved 7; // formerly legacy_partner_waiver, retired 2026-02-11
  RiskTier risk_tier = 8; // moved off the reused number
}

Second, make the schema reject this category of change at CI time instead of relying on a human reading a comment. The team already had buf in the toolchain for linting, just not wired into the pipeline that gates proto changes:

- name: Check for breaking proto changes
  run: |
    buf breaking --against 'https://github.com/org/protos.git#branch=main'

buf breaking flags field-number reuse as a breaking change by default, because it is one, even when the old field was never formally removed. This check alone would have failed the PR that introduced risk_tier = 7 before it ever shipped.

Third, an audit of every other message in the repo for deprecated-but-not-reserved fields. The grep was blunt but effective:

grep -rB1 "DEPRECATED" --include="*.proto" . | grep -v "reserved"

Thirty-one other fields across nine proto files matched: deprecated in a comment, never marked reserved, free to be silently reused by the next engineer who didn't read eight-month-old comments before picking a field number. All thirty-one got backfilled with reserved in the same cleanup PR.

the aftermath

All 228 affected orders were replayed against the current fraud model the same day. Most cleared on a second look, nine were flagged as genuinely high-risk and three of those were confirmed fraudulent, already past the chargeback window by the time the replay ran.

The shipping-cost rounding fix that triggered this deploy shipped clean. Nobody on that team touched fraud logic, reviewed fraud logic, or had any reason to think their rollout intersected with it at all. The two features were connected by nothing more than a field number neither team was actually watching.