How a Routine Node 14 to Node 18 Upgrade Turned a Six-Year-Old Bug Into a Crash Loop

09:14 UTC. PagerDuty: "inventory-sync: 0/4 tasks healthy." The deploy that shipped twenty minutes earlier wasn't a feature, it was a base image bump, Node 14 to Node 18, done purely to clear a security scan finding. No application code changed. The diff was eleven lines, all of them in a Dockerfile.

the scramble

First theory: the new Node image broke something in how the container starts. Node 18 changed its default OpenSSL provider, and there's a well-known class of "error:0308010C:digital envelope routines" crashes that hit older crypto code on the jump past Node 17. Someone greps the crash logs for that string. Nothing. The process isn't failing to start, it's starting fine, running for anywhere between ninety seconds and four minutes, then dying outright.

Second theory: memory. Node 18's default heap sizing behaves differently than Node 14's on the same container memory limit, and a tighter effective heap could plausibly OOM a service that was already close to its ceiling. kubectl describe pod says otherwise, exit code 1, not 137. Whatever killed these processes, it wasn't the kernel's OOM killer and it wasn't a Node heap limit. The process chose to exit.

Rolling back the image to Node 14 makes the crashes stop immediately, which rules out "unrelated bad luck" and confirms the upgrade really is the trigger. It doesn't yet explain how, since nobody touched the code the upgrade was supposedly isolated from.

the hunt

The crash logs all show the same two lines, which nobody had seen before because Node 14 never printed them for this code path:

node:internal/process/promises:289
    triggerUncaughtException(err, true /* fromPromise */);
    ^
Error: write ETIMEDOUT
    at TCP.onStreamRead (node:internal/stream_base_commons:216:20)
[... stack continues into warehouse-client.js:41 ...]
Node.js v18.18.2

warehouse-client.js:41 is a background sync loop that had existed unchanged since the service's first commit, six years earlier:

function startWarehousePoll() {
  setInterval(async () => {
    const updates = await fetchWarehouseDeltas(); // can reject on a slow upstream
    await applyToLocalCache(updates);
  }, 5000);
}

setInterval doesn't await the callback it's given and has no way to. If the async function inside it rejects, the promise that rejection belongs to was never returned to anything and never had a .catch() attached. It is, by definition, an unhandled rejection, every single time fetchWarehouseDeltas() hits a slow or dead upstream, which a warehouse API with its own maintenance windows did every few days.

That's not new. That exact rejection had been firing for years. What changed is what Node does about it:

Node 14: default mode = warn
  -> logs "UnhandledPromiseRejectionWarning" to stderr, process keeps running

Node 15+: default mode = throw
  -> the rejection becomes an uncaught exception, process exits

Node changed this default in Node 15, released back in 2020, specifically to stop silent promise rejections from hiding real bugs. Every LTS release since has kept throw as the default. The inventory-sync service had simply never crossed that version line until this upgrade, so a six-year-old latent bug had been quietly logging a warning line nobody read, on every slow warehouse API call, the entire time. The Node 18 jump didn't introduce a bug. It just stopped forgiving one that was already there.

The warehouse API's maintenance windows happening to land in the same hour as the deploy made the timing look causal in a way that made the Dockerfile diff the obvious, and misleading, first suspect.

the find

An async callback inside setInterval had no rejection handler and never had one. Node 14 treated that as a warning. Node 15 and every version after it treats the same rejection as a fatal, uncaught exception. The upgrade didn't break anything new, it just enforced a stricter default against six years of code that had always depended on the old, more forgiving one.

the fix

The immediate fix wraps the interval callback so a rejection becomes a logged, recoverable error instead of a process-ending one:

function startWarehousePoll() {
  setInterval(() => {
    pollOnce().catch((err) => {
      log.error('warehouse poll failed, will retry next tick', err);
      pollFailures.inc();
    });
  }, 5000);
}

async function pollOnce() {
  const updates = await fetchWarehouseDeltas();
  await applyToLocalCache(updates);
}

Second, a process-level safety net for the next one of these, since this was unlikely to be the only unattached promise in a six-year-old codebase:

process.on('unhandledRejection', (reason, promise) => {
  log.error('unhandled rejection, see stack below', reason);
  unhandledRejections.inc();
  // deliberately not exiting here: log and alert, let the
  // orchestrator's health checks decide if the process is actually unhealthy
});

Third, an ESLint rule that would have caught this years before any Node upgrade forced the issue:

rules: {
  '@typescript-eslint/no-floating-promises': 'error',
  '@typescript-eslint/no-misused-promises': 'error',
}

Both rules flag exactly this pattern, an async function handed to something that can't await it, with no explicit .catch(). Running it against the existing codebase after the incident turned up four more floating promises of the same shape, none of which had caused an outage yet only because their upstreams hadn't had a bad day since the Node 15 line was crossed.

the aftermath

Inventory sync being down for 22 minutes meant warehouse stock deltas queued instead of applying live; nothing was lost, the backlog drained in under two minutes once the fixed image rolled forward, and no order was accepted against stale inventory because the checkout service's own stock check hits a separate, unaffected read path.

The Dockerfile diff was eleven lines and genuinely had nothing wrong with it. The bug it exposed was committed years before anyone on the team that shipped this upgrade had joined.