How a Missing Workflow Version Gate Turned a Stuck Temporal Deploy Into 11 Duplicate Charges

10:14 UTC. The on-call channel for order-fulfillment gets a Temporal alert nobody on the team has seen fire before: workflow task failed: nondeterministic workflow definition, 23 times in six minutes, all from the same task queue. Nothing is down. The API is answering. Checkout is green. Somewhere underneath it, though, two dozen order workflows have simply stopped making progress, and the number is climbing.

the scramble

First theory: a worker crashed mid-execution and Temporal is retrying against a corrupted task. The worker fleet dashboard says otherwise, every pod healthy, CPU under 20%, no restarts in the last hour. Whatever's wrong isn't a dead process.

Second theory: Temporal Cloud is having a bad morning. The status page is clean, and temporal operator cluster health comes back SERVING in under a second. The platform is fine. Something this team owns is not.

A fulfillment order workflow is a saga: reserve inventory, charge the payment method, wait on a carrier-pickup signal that can take anywhere from minutes to two days, then mark the order shipped. That long wait is exactly why this incident doesn't look like a normal deploy problem. Whatever broke didn't break new orders. It broke orders that were already mid-flight when something changed underneath them.

the hunt

Pulling the history for one of the stuck workflow IDs is where the error stops being vague:

Status: Running
Last Event: WorkflowTaskFailed
Failure: "nondeterministic workflow definition: unexpected command
  ScheduleActivityTask(FraudCheck), expected command
  SignalExternalWorkflowExecution or StartTimer at history event 14"

The error names the mismatch directly: the code running right now wants to schedule a FraudCheck activity at a point in the workflow where this particular execution's recorded history has no such step. Checking the deploy log lines it up immediately. A change shipped at 09:52 UTC added a fraud-check activity call in the middle of the fulfillment workflow function, between the payment charge and the carrier-pickup wait, intended to catch high-risk orders before they ship:

func FulfillmentWorkflow(ctx workflow.Context, order Order) error {
  if err := workflow.ExecuteActivity(ctx, ReserveInventory, order).Get(ctx, nil); err != nil {
    return err
  }
  if err := workflow.ExecuteActivity(ctx, ChargePayment, order).Get(ctx, nil); err != nil {
    return err
  }

  // new: added 09:52 UTC, no version gate
  if err := workflow.ExecuteActivity(ctx, FraudCheck, order).Get(ctx, nil); err != nil {
    return err
  }

  workflow.GetSignalChannel(ctx, "carrier-pickup").Receive(ctx, nil)
  return workflow.ExecuteActivity(ctx, MarkShipped, order).Get(ctx, nil)
}

Every open workflow execution started before 09:52 has a recorded history that goes straight from ChargePayment to waiting on the carrier-pickup signal. There is no FraudCheck step in that history, because it didn't exist yet when the workflow started. Temporal replays a workflow's event history against its current code every time a worker picks it back up, to rebuild in-memory state without persisting it directly. That only works if the code is deterministic against whatever history already exists. A new code path inserted in the middle of an already-running saga, with no version gate, makes replay diverge from history on the first worker that happens to pick up an old workflow with new code. New workflows starting after 09:52 are unaffected; they generate fresh history that already matches the new code. It's only the ones caught mid-flight that break.

With the cause identified, the obvious-looking move is to just get the stuck orders moving again. One engineer tries exactly that on three of them:

$ temporal workflow terminate --workflow-id order-fulfillment-88213 --reason "stuck, restarting"
$ temporal workflow start --workflow-type FulfillmentWorkflow --workflow-id order-fulfillment-88213-retry --input order_88213.json

Terminating a stuck execution and starting a fresh one under a new workflow ID looks like a clean restart. It isn't. A brand-new execution has no history at all, which means every activity in the function body runs again from the top, including ReserveInventory and ChargePayment, both of which the original execution had already completed successfully before it ever got stuck. Temporal workflows don't remember what a different workflow ID already did. Three "restarted" orders meant three customers charged twice before anyone noticed the pattern, caught in time to stop a fourth.

the find

A workflow-definition change shipped without a version gate. That broke replay for every order workflow that was already mid-saga when the deploy went out. The actual root cause was entirely in application code, not in Temporal itself, and the terminate-and-restart response to it was the part that turned a stuck-workflow bug into a double-charged-customer bug, because restarting from zero re-executes every side-effecting activity a saga had already completed.

the fix

The workflow-code fix is workflow.GetVersion, which lets a single function definition branch its behavior based on what a given execution's history already contains, instead of assuming every replay sees the same code path:

func FulfillmentWorkflow(ctx workflow.Context, order Order) error {
  if err := workflow.ExecuteActivity(ctx, ReserveInventory, order).Get(ctx, nil); err != nil {
    return err
  }
  if err := workflow.ExecuteActivity(ctx, ChargePayment, order).Get(ctx, nil); err != nil {
    return err
  }

  v := workflow.GetVersion(ctx, "add-fraud-check", workflow.DefaultVersion, 1)
  if v == 1 {
    if err := workflow.ExecuteActivity(ctx, FraudCheck, order).Get(ctx, nil); err != nil {
      return err
    }
  }

  workflow.GetSignalChannel(ctx, "carrier-pickup").Receive(ctx, nil)
  return workflow.ExecuteActivity(ctx, MarkShipped, order).Get(ctx, nil)
}

On replay, GetVersion checks whether a version marker for "add-fraud-check" already exists in the execution's history. Old executions have none, so they get DefaultVersion and skip the fraud-check branch, matching exactly what they already did. New executions record the marker and take the new branch. One function definition, two valid histories.

Cleaning up the mess the restarts caused needed a separate pass. A script cross-referenced every workflow ID terminated between 10:14 and 10:31 UTC against the payment gateway's charge log by order ID and flagged any order with two successful charges in that window. Three orders matched. All three got the duplicate charge refunded within the hour, before any customer had to notice or file a dispute.

the aftermath

Once the versioned workflow code redeployed, all 23 stuck executions resumed on their own within the next worker poll cycle, no manual intervention needed for the 20 that nobody had touched. Zero data loss on any of them, since replay failures leave the recorded history exactly as it was; they just can't move forward until the code agrees with it again. The three manually restarted orders were the only ones that needed a cash refund rather than a code fix.

Nothing about this incident was a platform failure. Temporal did exactly what it's designed to do: refuse to silently guess at a workflow's state when the code and the history disagree. The cost came entirely from the fix that skipped past that refusal instead of listening to it.