October 18, 2026Architecture8 min read

How a Forward-Compatible Schema Change Deleted a Kafka Field and Zeroed $9,140 in Reimbursements

Published October 18, 20268 min read

02:14 UTC. The nightly reconciliation job finishes clean, exit code 0, same as it has every night for three years. Nobody gets paged for it, there's nothing to page on. Six hours later, someone on finance opens the carrier reimbursement report and the number that's supposed to clear $10,000 reads $187.

the scramble

Finance pings #eng-oncall assuming it's a reporting bug, not a six-hour data loss. First theory: a bad join dropped rows somewhere in the warehouse transform. The on-call engineer reruns the report's SQL by hand against yesterday's partition. Same $187. Counts the source rows directly, 2,380 orders shipped overnight, which matches the carrier's own portal count exactly. The rows are all there. Nothing got dropped.

Second theory, and the one that feels more likely given the number: the carrier's API started returning $0 for every return-label cost lookup, maybe a billing dispute, maybe an API key that rotated badly. Someone pulls a dozen order IDs from last night's batch and calls the carrier's pricing endpoint directly for each one. Real numbers come back. $11.40, $9.80, nothing close to zero. The carrier is fine.

Twenty minutes in, both obvious theories are dead, and the report itself isn't miscalculating anything. It's faithfully summing whatever is already sitting in the warehouse table. Something upstream of the report put a zero there on purpose, or something that looks like purpose.

the hunt

The reconciliation job reads from a Kafka topic, order.shipped, Avro-encoded, one event per shipment, consumed once a night by a Python batch using confluent-kafka's AvroDeserializer against the company's Confluent Schema Registry. Instead of trusting the warehouse table, pull the raw decoded events straight from the topic and compare last night's batch against a batch from three days ago.

decoded events, three days apart

# three nights ago
{"order_id": "ORD-88213", "carrier": "ups", "return_shipping_cost_cents": 940}

# last night
{"order_id": "ORD-91047", "carrier": "ups", "return_shipping_cost_cents": 0}

Every event from last night carries return_shipping_cost_cents: 0, not missing, not null, a clean zero, consistent with a schema default kicking in rather than a parsing failure. That rules out a batch job bug. The zero is baked into what the topic hands back once decoded, and it started at a specific point in time. Checking the schema registry's version history for the subject confirms it.

subject version history

curl -s http://schema-registry.internal:8081/subjects/order.shipped-value/versions
[9, 10, 11, 12, 13]

curl -s http://schema-registry.internal:8081/subjects/order.shipped-value/versions/13 | jq '.schema' | jq .
# no "return_shipping_cost_cents" field anywhere in the record

curl -s http://schema-registry.internal:8081/subjects/order.shipped-value/versions/12 | jq '.schema' | jq .
# {"name": "return_shipping_cost_cents", "type": "long", "default": 0}

curl -s http://schema-registry.internal:8081/config/order.shipped-value
{"compatibilityLevel": "FORWARD"}

Version 13 was registered at 14:02 UTC the previous day, by the shipping service's deploy. It deletes return_shipping_cost_cents entirely. Version 12, the schema the nightly batch job's deserializer is still pinned to, has that same field with default: 0. The compatibility level on this subject is FORWARD. Those three facts together are the whole bug.

the find

FORWARD compatibility guarantees that data written with the new schema can still be read using the previous schema. The team set this subject to FORWARD two years ago, around the same time the carrier integration was built, so the shipping service could keep shipping schema changes without waiting on every downstream consumer to redeploy first. Reasonable trade-off at the time. For a field deletion specifically, the registry allows it as long as the previous (reader) schema already has a default for that field, because that default is exactly what lets the old reader fill the gap when the field is simply absent from new data. Version 12's default: 0 on return_shipping_cost_cents wasn't there for this reason. It was added two years earlier, during the original carrier integration, when the field was briefly optional while that pipeline was still being built out. Once real per-shipment costs started flowing in, nobody went back and removed the now-pointless default. It just sat there, unused, for two years.

When the shipping service's maintainer deleted the field in version 13, cleaning up a value that had migrated to a separate carrier-cost service months earlier, the registry's compatibility check saw version 12's dormant default and approved the change instantly. The CI schema-check step went green exactly as it was designed to. Every event produced after that deploy stopped carrying the field. The nightly batch job's deserializer, still pinned to version 12 because nothing had ever forced it to bump, filled in the missing field with version 12's own default on every single message. It wasn't a bug in the deserializer. It did precisely what the schema told it to do, applied correctly to an event that was quietly incomplete.

the fix

Recovery for the six-hour window came first. The carrier keeps its own authoritative per-label cost on their side regardless of what made it into the topic, so the real numbers got pulled from their shipment-cost export and backfilled directly into the warehouse table for the affected orders.

The root cause took a bit more work. The nightly batch's deserializer got bumped to version 13, and the reimbursement figure it needs now comes from the carrier-cost service's own topic, which is where that number has genuinely lived since the migration months ago. Nothing downstream should have still depended on a two-year-old default standing in for a real value.

What actually stops this from happening again is a pre-merge check that now runs against every schema change on this registry. It diffs the proposed schema against the currently registered one and flags any field deletion where the existing reader schema still carries a default, since that's the exact shape of change a FORWARD compatibility check will pass without complaint.

schema-compat-guard.py, run in CI before registration

def find_risky_deletions(old_schema: dict, new_schema: dict) -> list[str]:
    old_fields = {f["name"]: f for f in old_schema["fields"]}
    new_field_names = {f["name"] for f in new_schema["fields"]}
    deleted = [name for name in old_fields if name not in new_field_names]
    return [name for name in deleted if "default" in old_fields[name]]

if __name__ == "__main__":
    risky = find_risky_deletions(load_schema(OLD_VERSION), load_schema(NEW_VERSION))
    if risky:
        print(f"BLOCKED: deleting {risky} passes FORWARD compatibility "
              f"silently via an existing default on the reader schema")
        sys.exit(1)

A daily reconciliation alert was added too, comparing each day's total reimbursement against a rolling 14-day median and paging on-call if it drops more than 40%, instead of waiting for finance to notice a number looks wrong in a report.

the aftermath

6 hrs Window of events produced after the deploy with the field silently zeroed

2,380 Orders affected before the deserializer was bumped off the stale schema version

$9,140 Reimbursement total recovered from the carrier's own shipment-cost export

40% Drop from the 14-day median that now pages on-call automatically

FORWARD compatibility doesn't mean a field is safe to delete. It means some reader schema can fill the gap, and that gap-filler can be a default nobody remembers setting, doing nothing for years until the day it quietly starts doing everything.
A deserializer applying a default isn't malfunctioning. It's doing its job correctly against an input that's wrong two layers upstream, which is exactly why the failure doesn't show up as an error anywhere in the pipeline.
A batch job finishing with exit code 0 tells you it ran. It tells you nothing about whether it ran on correct data, and those are different claims that only one of them is checked by default.
A compatibility mode chosen for the agility it buys a producer team is also a standing policy about what consumers are allowed to silently not notice. Two years is long enough for everyone who understood that trade-off to have moved teams.

Nobody downstream did anything wrong. The carrier billed correctly, the report summed its inputs correctly, the deserializer applied its default correctly. The actual break sat two systems upstream, in a compatibility mode and a default value that had been configured correctly too, just never configured to work together once one of them changed.