How an Unpinned Base Image Tag Let V8 Size Its Heap Off the Host, Not the Pod

14:47 UTC, Tuesday. Pods in the checkout-api deployment start OOMKilling in a loop, three or four at a time, in the middle of an ordinary afternoon. Traffic is flat. No deploy went out today. PagerDuty fires on KubePodCrashLooping, not on anything tied to a release, because as far as the deploy history shows, nothing was released.

the scramble

First theory: a memory leak shipped yesterday and took a day to build up enough pressure to matter. git log --since=48h -- apps/checkout-api shows exactly one commit, a routine lockfile refresh from the nightly dependency job. No application code changed in either direction.

Second theory: a traffic burst nobody's dashboard caught. Request rate for the service is within 3% of the same Tuesday last week. Whatever's killing these pods, it isn't load.

Third theory: a slow leak finally crossing a threshold. The service exposes a /debug/memory endpoint that dumps process.memoryUsage() on demand. heapUsed sits at a boring 140MB, same as it's sat for weeks. rss, meanwhile, is climbing steadily past 480MB on a pod with a 512Mi limit and hasn't leveled off. Whatever's growing, V8's own heap accounting doesn't think it's growing.

the hunt

A gap between heapUsed and rss usually means something outside V8's managed heap: a native addon, an unreleased buffer, a leaking file descriptor. That's the next hour, chasing a native leak. clinic doctor against a staging replay shows nothing unusual. No native module in the dependency tree changed in the lockfile refresh. Dead end.

The clue that actually breaks it open is which pods die fastest. Traffic is round-robin across all pods, cgroup memory limits are identical at 512Mi, and yet pods scheduled onto the cluster's larger nodes survive noticeably longer before OOMKilling than pods on the smaller ones. That shouldn't matter. A container's enforced limit is supposed to be the same number regardless of what the underlying node has installed.

# pod on a 16GB node
$ kubectl exec checkout-api-7f9d-a1 -- node -e \
  "console.log(require('v8').getHeapStatistics().heap_size_limit / 1e6)"
2187.4

# pod on a 64GB node
$ kubectl exec checkout-api-7f9d-b2 -- node -e \
  "console.log(require('v8').getHeapStatistics().heap_size_limit / 1e6)"
8341.9

Neither number is close to 512. Both are absurd for a container capped at 512Mi, and they scale with the node's physical memory, not with the pod's cgroup limit, which is the opposite of what should happen. A canary pod still running the previous image, from before last night's rebuild, gives a third data point:

$ kubectl exec checkout-api-canary-prev -- node -e \
  "console.log(require('v8').getHeapStatistics().heap_size_limit / 1e6)"
462.8

462MB, correctly tracking the 512Mi limit regardless of which node it lands on. The canary and the failing pods are running the same application code. The only thing that differs is the image.

$ docker inspect checkout-api:canary-prev --format '{{.Config.Image}}'
node:18.19.1-alpine

$ docker inspect checkout-api:latest --format '{{.Config.Image}}'
node:18.20.4-alpine

The Dockerfile's first line has always read FROM node:18-alpine, a floating tag, never a pinned version. The nightly CI job that rebuilds this image runs docker build --pull, which fetches whatever node:18-alpine currently resolves to on every run. Somewhere between two nights ago and last night, that tag moved from 18.19.1 to 18.20.4. No commit, no changelog line, no PR. The base image changed underneath the Dockerfile without the Dockerfile changing at all.

the find

Root cause: Node determines its default heap ceiling by probing the container's cgroup memory limit through libuv, and falling back to the host's total memory if that probe fails. The 18.20.4 alpine layer repackaged its cgroup compatibility shims in a way that changed which path libuv checks first on this cluster's cgroup v2 nodes. On 18.19.1 the probe found memory.max and sized the heap off 512Mi, as intended. On 18.20.4 the probe missed and silently fell back to os.totalmem(), which reports whatever the underlying node actually has, 16GB on some nodes and 64GB on others.

V8 isn't wrong about the number it was given. It's sizing its default old-space target off a number that has nothing to do with what the container is actually allowed to use. GC pressure heuristics only kick in as the heap approaches its perceived limit, so with a limit in the gigabytes, V8 kept committing more memory well past 512Mi, confident it had plenty of room left. The kernel's cgroup OOM killer doesn't consult V8's opinion. It watches RSS against the pod's actual 512Mi ceiling and sends SIGKILL the moment that's crossed, regardless of what heapUsed reports.

the fix

Pin the base image to a digest, not a tag, so a rebuild can never silently change what ships:

# before
FROM node:18-alpine

# after
FROM node:18.19.1-alpine@sha256:4a7a4e52c40e973c3e4b02f0c9d8d6c7f8e1a0b2c3d4e5f6a7b8c9d0e1f2a3b4

Second, stop letting V8 guess at a limit it can get wrong. The pod's memory limit is known at build time via Kubernetes resource specs, so the heap ceiling gets set explicitly instead of auto-detected:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=384"
resources:
  limits:
    memory: "512Mi"
  requests:
    memory: "512Mi"

384MB leaves roughly 25% of the 512Mi limit for stack space, native allocations, and whatever else lives outside V8's managed heap, the same gap that let rss outrun heapUsed during the incident. With the ceiling fixed explicitly, a future base image change can't move it again.

Third, the CI job that builds this image now fails loudly on an unpinned FROM line instead of silently rebuilding against whatever a tag resolves to that day:

if grep -qE '^FROM [a-z0-9./-]+:[a-zA-Z0-9._-]+$' Dockerfile && \
   ! grep -q '@sha256:' Dockerfile; then
  echo "FROM line is not pinned to a digest, failing build"
  exit 1
fi

the aftermath

Nothing in this incident involved a bug in the application. The code that ran at 14:47 UTC was identical to the code that had run cleanly for weeks. What changed was three layers down, in a base image tag nobody was watching, and it took exec-ing into pods on different physical nodes to even notice the one variable that should never have mattered was the variable that did.