How MongoDB's Default Write Concern Rolled Back 266 Confirmed Orders During a Replica Set Election

02:14 UTC. Zendesk lights up with eleven tickets in nine minutes, same shape: "I got a confirmation email, Stripe charged my card, but your order history says I don't have an order." The on-call engineer's first move is the Stripe dashboard. Every charge is real, captured, settled. The orders just aren't in the database.

the scramble

The obvious suspect is the maintenance window AWS had scheduled six hours earlier for the underlying host of one node in the orders replica set, three members across two AZs. That ticket is already closed, marked no-impact: request success rate held at 100% through the window, error rate dashboards flat, the replica set elected a new primary and kept serving traffic without a single failed write. Textbook failover. Nobody reopens a ticket that says everything worked.

Second theory: a client-side double-submit bug sending confirmation emails for orders that never actually got created. Ruled out fast, the confirmation email template only fires server-side, after the insert into orders returns success. If the email went out, the write was acknowledged. Something acknowledged isn't supposed to just stop existing afterward.

Third theory, the one that goes somewhere: someone plots the affected order timestamps against the maintenance window and they land in a seven-second band right at the start of it, 02:14:03 to 02:14:10 UTC. Not scattered across the whole maintenance period. One tight window, at the exact moment the failover happened.

the hunt

The checkout service's insert path looks ordinary. It was tuned six weeks earlier, ahead of a product launch, specifically to shave write latency:

const result = await db.collection('orders').insertOne(orderDoc, {
  writeConcern: { w: 1 }, // changed from default 'majority' for latency, 6 weeks ago
});

await kafka.send({
  topic: 'order.confirmed',
  messages: [{ value: JSON.stringify(orderDoc) }],
});

await sendConfirmationEmail(orderDoc.customerEmail, orderDoc._id);
return { orderId: orderDoc._id, status: 'confirmed' };

w: 1 means the primary acknowledges the write the moment it's durable on its own disk, before any secondary has it. That's the latency win: no waiting on cross-AZ replication round trips. It's also the whole story, once you know to look for it. The insert returning success only means the node that was primary at that instant has the document. It says nothing about whether that node stays primary.

Grepping the primary's own log for what happened in that seven-second window:

$ grep -h "REPL\|ROLLBACK" /var/log/mongodb/mongod.log | sed -n '/02:14:0[0-9]/,/02:14:12/p'
{"t":"2026-10-31T02:14:03.102Z","c":"REPL","msg":"Could not find member to sync from"}
{"t":"2026-10-31T02:14:03.104Z","c":"REPL","msg":"Standing for election","attr":{"reason":"heartbeat timeout"}}
{"t":"2026-10-31T02:14:09.881Z","c":"REPL","msg":"Replica set state transition","attr":{"newState":"SECONDARY"}}
{"t":"2026-10-31T02:14:09.912Z","c":"ROLLBACK","msg":"Starting rollback","attr":{"lastCommonPointOpTime":"02:14:02.998Z"}}
{"t":"2026-10-31T02:14:11.236Z","c":"ROLLBACK","msg":"Rollback summary","attr":{"docsDeleted":266,"namespace":"ecommerce.orders"}}

The maintenance host reboot didn't touch the primary directly. It touched a secondary in az-2, and a route table change that shipped in the same window cut cross-AZ connectivity between the primary and both other members for roughly eight seconds. The two az-2 members, unable to reach the primary, held an election between themselves once their heartbeat timeout fired and picked a new primary. The old primary, still physically healthy and still accepting local writes, hadn't yet detected it had lost the majority. For those seven seconds it was a primary with no followers, happily acknowledging w: 1 inserts that no other node would ever see.

When it finally noticed, it stepped down and rejoined as a secondary. Syncing with the new primary, it found its own oplog had entries past the last point the two oplogs agreed on, 02:14:02.998Z, entries the new primary had never received. Per MongoDB's rollback algorithm, those entries get reverted: the affected documents are written out to BSON files under the data directory's rollback/ folder, then deleted from the live collection to bring it back in sync with the new primary's history.

$ ls /var/lib/mongo/rollback/ecommerce.orders.*.bson
/var/lib/mongo/rollback/ecommerce.orders.3f9a1c2e-88b1-4a47-9c3e-2d7a6b901234.bson

$ bsondump /var/lib/mongo/rollback/ecommerce.orders.3f9a1c2e-*.bson | jq -r '._id' | wc -l
     266

$ bsondump /var/lib/mongo/rollback/ecommerce.orders.3f9a1c2e-*.bson | jq -r '._id' | sort > rolled_back.txt
$ comm -12 rolled_back.txt zendesk_affected_order_ids.txt | wc -l
     266

the find

266 documents, exact match against the Zendesk list. The replica set hadn't lost a single request, by the metric anyone was watching. It had lost something else: the guarantee that an acknowledged write stays acknowledged once the node that acknowledged it stops being the primary. w: 1 trades that guarantee for latency, on purpose, and that trade is fine for data you can afford to lose. Nobody had checked whether "confirmed, charged, emailed" still qualified once the setting moved off its default six weeks earlier.

the fix

The fix is putting the write concern back to majority for this one write path, and only this one. w: 1 stays in place for writes that genuinely don't need the guarantee, like updating a product view counter, where losing a few increments during a once-a-year network blip is a non-event.

const result = await db.collection('orders').insertOne(orderDoc, {
  writeConcern: { w: 'majority', wtimeout: 5000 },
});
// now blocks until a majority of the set has the write durably,
// so a step-down can't un-happen it out from under us

w: 'majority' only returns once at least two of the three members have the write on disk, which means the write exists on whichever node ends up winning any subsequent election. A primary can still lose majority contact and step down, but it can't acknowledge a write that the new primary doesn't also have. The p50 write latency for order creation went from 8ms to 17ms. The p99 went from 22ms to 61ms, almost entirely cross-AZ round trip. Both numbers were acceptable for a write that happens once per order, not once per request.

Recovering the 266 orders turned out to need more than just replaying the BSON files. Fifteen of them had already been swept by a separate, unrelated disk-space cron that purges anything under /var/lib/mongo/ older than 24 hours, a job nobody on the database team had cross-checked against what lives in rollback/. Those fifteen got reconstructed by joining the order.confirmed Kafka events, which still had the full order payload, against Stripe's captured-charge records for the same customer and amount. The other 251 came straight back from the BSON files with a one-off restore script.

the aftermath

No customer lost money and no charge went unfulfilled, but it took 51 minutes from the first Zendesk ticket to the last order restored, and all of it was avoidable if the write concern change six weeks earlier had come with a one-line question: what happens to this specific write during an election.

The maintenance window really was handled correctly, in the sense the team originally meant: the cluster stayed up, requests kept succeeding, nothing timed out. That was always the wrong question. Availability was never what broke.