How a Missing Concurrency Group Let a Slower Build Undeploy a Hotfix Nine Minutes After It Shipped

10:52 UTC. The checkout error rate graph flatlines at zero, right where everyone wants it after a hotfix. Someone posts a checkmark emoji in #incidents. At 11:01 UTC the same graph is back above 4%, the exact shape it had before the fix went out, like the last eleven minutes never happened.

the setup

Checkout had been throwing discount_code is undefined for about forty minutes, traced to a null check that got dropped in a refactor merged the night before. A one-line hotfix went up as its own PR, reviewed fast, merged straight to main at 10:47 UTC.

Deploys ran off a standard workflow: push to main triggers a build, then a deploy job ships whatever the build produced to production.

name: Deploy
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/

  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: build-output
          path: dist/
      - run: ./scripts/deploy-prod.sh

Nothing in that file limits how many of these can run at once. Every push to main gets its own independent run, start to finish, and there was no reason to think that mattered, because merges to main were usually spaced minutes apart.

They weren't spaced far enough apart this time. At 10:44 UTC, three minutes before the hotfix merged, an unrelated PR, a larger change to the search indexing pipeline that had been sitting in review since the day before, also merged to main. Its build pulled in a dependency update that added about eleven minutes to npm run build. The hotfix's build, one line of code, no new dependencies, took three minutes.

the scramble

First theory: a CDN cache serving a stale bundle. The hotfix had deployed successfully, checkout worked, and then it stopped working again with the same error, which reads like something got served from a cache instead of the new deploy. Someone purged the Cloudflare cache for the checkout routes.

on-call: purging CF cache for /checkout/* just in case, error rate
          looks identical to before the fix so it might just be stale

The purge didn't change anything. The error was still there two minutes later, at the same rate.

Second theory: the hotfix PR hadn't actually fixed the bug, and the first all-clear was a fluke, maybe a brief window with no discount codes in the traffic mix. That one fell apart fast. Someone reran the exact request that had been failing before 10:52 UTC against the current production build using curl, and it still threw the same error, meaning whatever was live was not the code that had been reviewed and merged an hour ago.

Dead end three: check whether the deploy script itself had failed partway and left production in a mixed state. The GitHub Actions run for the hotfix commit showed green, start to finish, both jobs. Nothing there looked broken.

the hunt

The health endpoint reports the git SHA baked in at build time. That was the fastest way to answer the only question that mattered: what commit is actually running right now.

$ curl -s https://api.example.com/health | jq .commit
"a3f9e21"

$ git log -1 --format=%h a3f9e21
a3f9e21 Merge PR #4182: rework search indexing batch size

$ git log -1 --format=%h HEAD
7c14bd8 Merge PR #4189: fix undefined discount_code in checkout

Production was running a3f9e21, the search indexing merge from before the hotfix, not 7c14bd8, the hotfix itself. The GitHub Actions run history made the sequence obvious once someone thought to sort by finish time instead of start time.

run  commit    started    completed   duration
#412 7c14bd8   10:47:03   10:51:40    4m 37s   <- hotfix, deployed first
#411 a3f9e21   10:44:11   11:00:52    16m 41s  <- search PR, deployed second

Both runs had triggered off main within three minutes of each other. Both ran the build job and the deploy job entirely independently, with no coordination between them. Run #412, the hotfix, had less work to do and finished first, so it deployed first, and the error rate correctly dropped to zero. Run #411 kept building in the background, for a change that had nothing to do with checkout, and when it finished sixteen minutes later, its deploy-prod.sh step did exactly what every previous deploy had always done: shipped the artifact from its own build to production. It had no idea a newer commit had already deployed after it started, or that overwriting it would silently reintroduce a bug.

the find

Root cause: the deploy workflow had no concurrency control, so two runs for two different commits were allowed to execute in parallel and reach production in whatever order their builds happened to finish, not the order the commits landed in. A workflow that deploys "whatever this run built" instead of "whatever the latest commit is" will silently regress production the moment a slower build for an older commit outlives a faster build for a newer one.

It's a timing coincidence that this surfaced now. Every previous pair of nearby merges had happened to resolve build times in commit order by luck. The search indexing PR's dependency bump was the first thing to stretch a build long enough to lose that race, and a one-line hotfix merging three minutes later was exactly the kind of commit fast enough to win the race to production and then lose it again once the slower run caught up.

the fix

Two changes went in together. The first: a concurrency group on the deploy workflow, so a new run cancels whatever's still in flight for the same target instead of racing it.

name: Deploy
on:
  push:
    branches: [main]

concurrency:
  group: deploy-production
  cancel-in-progress: true

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/

  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: build-output
          path: dist/
      - run: ./scripts/deploy-prod.sh

cancel-in-progress: true means run #411 would have been cancelled the moment run #412 started, instead of quietly finishing sixteen minutes later and shipping stale code. That alone closes the hole for this specific race. It doesn't cover every path to production, though, so a second, cheaper guard went into deploy-prod.sh itself: refuse to deploy if the commit being deployed isn't a descendant of whatever's currently live.

LIVE_SHA=$(curl -s https://api.example.com/health | jq -r .commit)
DEPLOY_SHA=$(git rev-parse HEAD)

if [ "$LIVE_SHA" != "$DEPLOY_SHA" ] && \
   ! git merge-base --is-ancestor "$LIVE_SHA" "$DEPLOY_SHA"; then
  echo "refusing to deploy: $DEPLOY_SHA is not a descendant of live commit $LIVE_SHA"
  exit 1
fi

./deploy.sh dist/

That guard catches cases the concurrency group won't, like a manually re-triggered old run, a rollback script invoked out of order, or a second CI provider entirely. It costs one curl call and one git check per deploy.

the aftermath

Nothing in this incident threw an error anyone was watching for. Both workflow runs succeeded. Both deploys completed without complaint. The bug came back because the pipeline had no concept of commit order, only of run completion order, and those two things are only the same by coincidence once builds vary enough in length.

The concurrency group would have prevented this exact race. The ancestry guard in the deploy script is there for every race it wouldn't.