How Force-Unlocking a Stale Terraform Lock Blackholed Our Checkout Egress

11:52 UTC. PagerDuty: "checkout-worker: Stripe API error rate 85%, 3 min." No deploy to checkout in the last five days. No incident on Stripe's status page. Whatever broke, it broke underneath the service, not inside it.

the setup

All networking for the production account, VPCs, subnets, NAT gateways, route tables, security groups, lives in one Terraform workspace: prod-network. State sits in S3, locking goes through a DynamoDB table, and CI applies it on every merge to main. It's the kind of setup that made sense when the account had one VPC and a handful of security groups, and nobody had gotten around to splitting it since.

That morning, a merged PR was mid-flight: move the private subnets' NAT gateway from us-east-1a to us-east-1c for AZ redundancy. The plan was straightforward, create a new gateway, point the private route table's default route at it, then remove the old gateway's resource block so Terraform would tear it down.

resource "aws_eip" "natgw_new" {
  domain = "vpc"
  tags   = { Name = "natgw-us-east-1c-eip" }
}

resource "aws_nat_gateway" "new" {
  allocation_id = aws_eip.natgw_new.id
  subnet_id     = aws_subnet.public_c.id
  tags          = { Name = "natgw-us-east-1c" }
}

resource "aws_route" "private_egress" {
  route_table_id         = aws_route_table.private.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id          = aws_nat_gateway.new.id
}

# aws_nat_gateway.old and aws_eip.natgw_old_eip removed from config in this PR

CI kicked off the apply at 11:02 UTC. Plan: one to add, one to change, one to destroy.

the scramble

First theory: Stripe. Their status page was fully green, no incidents, no degraded components. Dead end within thirty seconds.

Second theory: a security group change. A different PR, unrelated to networking, had merged about an hour earlier: add an outbound rule allowing traffic to a new fraud-detection vendor's IP range. On-call pulled the diff. It was exactly what it said, one egress rule, one CIDR block, nothing touching NAT or routing. Dead end.

Third theory came from actually looking at the route, not guessing around it.

$ aws ec2 describe-route-tables --route-table-ids rtb-0a1b2c3d \
    --query 'RouteTables[0].Routes'

[
  {
    "DestinationCidrBlock": "0.0.0.0/0",
    "NatGatewayId": "nat-0f7e8a2b1c9d4e5f6",
    "State": "blackhole"
  }
]

blackhole. The default route for every private subnet pointed at a NAT gateway that AWS no longer considered valid, and had been silently dropping every packet that hit it since the moment that gateway stopped existing.

the hunt

CloudTrail for the last hour, filtered to the NAT gateway's resource ID, told the actual story.

11:52:03 DeleteNatGateway    nat-0f7e8a2b1c9d4e5f6  (state: deleting)
11:52:04 ReplaceRoute        rtb-0a1b2c3d           (pending, waiting on new target)
11:56:25 DeleteNatGateway    nat-0f7e8a2b1c9d4e5f6  (state: deleted)
11:56:26 ReplaceRoute        rtb-0a1b2c3d           (target: nat-0a9b8c7d6e5f4a3b, completed)

The old gateway's delete call went out at 11:52:03. The route pointing at it flipped to blackhole the same second, because AWS marks a route as invalid the moment its target starts tearing down, not when teardown finishes. NAT gateway deletion isn't instant, AWS keeps it in a deleting state for a couple of minutes before it's actually gone, and Terraform's aws_nat_gateway resource waits for that full deletion before it considers the destroy complete. The route update that should have pointed traffic at the new gateway didn't run until 11:56:26, four minutes and twenty-three seconds after the old one started dying.

But this apply was supposed to be the fraud-detection security group rule. Pulling the actual CI job log for that run answered why a one-line SG change had touched NAT and routing at all.

11:47:11 Acquiring state lock. This may take a few moments...
11:47:11 Error: Error acquiring the state lock

Lock Info:
  ID:        c4f1a9e2-7b3d-4e6a-9f2c-1d8e6a4b7c9d
  Path:      prod-network/terraform.tfstate
  Operation: OperationTypeApply
  Who:       ci-runner-a3f8@gh-actions
  Created:   2026-08-29 11:02:14 UTC

11:50:02 [manual] terraform force-unlock c4f1a9e2-7b3d-4e6a-9f2c-1d8e6a4b7c9d
11:51:40 Plan: 2 to add, 1 to change, 1 to destroy
11:52:01 Apply complete after 4 approvals

The 11:02 job, the NAT migration, had been holding the lock for 45 minutes. Its CI dashboard entry still showed "running", because the runner pod had been OOM-killed by a co-scheduled job spiking memory on the same node at 11:04, two minutes after it started. A SIGKILL never gets the chance to release a Terraform lock or report failure back to CI, so the job just sat there, looking alive.

On-call saw a stuck-looking job blocking an urgent fraud rule, and force-unlocked it. The next apply computed a fresh plan against current state and current config, and current config already had the NAT migration merged in from three hours earlier. Since the workspace applies everything in one shot, that plan bundled the fraud rule with the NAT gateway's create, the route's update, and the old gateway's destroy, all in a single approval. The plan output listed all four changes. Nobody scanned past the one they were there for.

the find

Root cause: the route referenced the new gateway's ID, and the old gateway's config block was simply gone, so nothing in Terraform's dependency graph tied the route update to happening before the old gateway's destroy. Terraform doesn't infer an ordering constraint between two resources unless one references the other's attributes, and here neither did. In this apply, the destroy landed first in the execution order, and because NAT gateway deletion is a multi-minute AWS operation rather than an instant one, every private-subnet request that needed a fresh connection during that window hit a blackholed route with no gateway behind it.

The stale lock and the force-unlock were how an unrelated, low-risk PR ended up carrying a live-traffic infrastructure cutover it never should have shared an apply with. The ordering bug in the NAT migration itself was always going to fire eventually, on whatever apply happened to run it. The shared workspace just decided that apply would be a security group tweak from someone who had no reason to review a NAT gateway diff at all.

the fix

The migration pattern itself was the real bug. Creating a replacement resource, cutting traffic over to it, and destroying the original all in one apply is the same mistake as an expand-contract database migration done in one step instead of two. We now require these as separate applies:

resource "aws_nat_gateway" "new" {
  allocation_id = aws_eip.natgw_new.id
  subnet_id     = aws_subnet.public_c.id
  tags          = { Name = "natgw-us-east-1c" }
}

resource "aws_route" "private_egress" {
  route_table_id         = aws_route_table.private.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id          = aws_nat_gateway.new.id
}

# aws_nat_gateway.old stays declared, untouched, until traffic is verified healthy
# on the new gateway. It is not removed from config in this PR.

A second PR, merged only after confirming egress health on the new gateway, removes the old block. That apply touches exactly one resource, an idle NAT gateway nothing depends on anymore, so even a worst-case force-unlock on it can't take checkout down.

Second, the lock table now carries enough information to tell a stuck job from a dead one before anyone reaches for force-unlock.

import boto3

table = boto3.resource("dynamodb").Table("terraform-locks")

def check_stale_locks(max_age_minutes=10):
    for item in table.scan()["Items"]:
        age = current_lock_age_minutes(item)
        if age > max_age_minutes:
            runner_alive = check_ci_runner_status(item["Who"])
            if not runner_alive:
                alert_slack(
                    f"Lock {item['LockID']} on {item['Path']} is {age}min old, "
                    f"owning runner is dead. Safe to force-unlock after review."
                )

This is the difference that mattered most. Instead of on-call inferring "stuck" from a spinning CI dashboard, they'd get a direct answer: the runner is confirmed dead, here is the lock ID, and here is a reminder to review the pending plan before applying it.

Third, prod-network is being split. Routing and NAT live in one state now, security groups in another. An urgent SG change should never again be physically capable of carrying somebody else's half-finished infrastructure migration into production.

Fourth, a canary now curls an external endpoint from inside each private subnet every 30 seconds and alerts on failure. It would have caught this blackhole within half a minute of it starting, instead of waiting on checkout's own customer-facing error rate to cross a threshold three minutes in.

the aftermath

By the time CloudTrail confirmed the root cause, Terraform's own apply had already finished the route update it was going to run anyway, four minutes after the destroy, and checkout had already recovered on its own. The investigation wasn't racing to fix the symptom. It was racing to understand why it happened before the same ordering gap fired on some future apply nobody would be watching as closely.

The lock-watchdog has flagged two more stale locks since, both from the same class of OOM-killed runner. Both got a reviewed force-unlock and a clean, isolated apply. Neither made it anywhere near checkout.