How Restarting 40 Pods at Once Filled a Node's Conntrack Table and Broke Services That Were Never Touched

15:04 UTC. Three unrelated services start throwing connection errors within the same ninety-second window: billing-api can't reach Postgres, the image-resize worker can't reach Redis, and the internal search proxy starts timing out on calls to a service two hops away. Nobody had deployed any of those three in the last six hours. The only thing that had shipped recently was a config change to notifications-service, a service none of the three affected teams had ever heard of.

the setup

notifications-service runs 40 replicas on a shared EKS node pool, general-purpose worker nodes that also host billing-api, the image-resize worker, and a dozen other unrelated services. Nothing about that is unusual, bin-packing workloads onto shared nodes is the entire point of running Kubernetes instead of one VM per service.

That afternoon, an engineer pushed a small env var change to notifications-service and ran kubectl rollout restart deployment/notifications-service to pick it up. With the deployment's default rolling strategy, that meant roughly ten pods cycling at a time in overlapping waves, most of the fleet turned over within about ninety seconds. The node pool backing this workload only has six nodes, so across the full restart, several rounds of new pods cycled through each one, including node ip-10-4-21-88, a normal outcome of the scheduler's bin-packing, not anything the engineer configured on purpose.

Every notifications-service pod, on startup, opens a ten-connection pool to Postgres through PgBouncer, connects to Redis for its dedup cache, and calls an internal billing-api client to register the fresh replica. All three of those are short Kubernetes service names, pgbouncer, redis, billing-api, resolved through CoreDNS. Nobody had looked at that node's nf_conntrack table in months. It had quietly been sitting around 63,700 of a 65,536-entry limit, the sysctl default the node's AMI shipped with, never tuned for how many pods this cluster actually schedules per node.

the scramble

First theory, from the billing-api on-call: Postgres itself. The error was a connection failure, so the obvious suspect was the database running out of connections or PgBouncer falling over. SHOW POOLS on PgBouncer showed normal utilization, nowhere close to its limit, and Postgres's own connection count hadn't moved.

Second theory, from the image-resize team: a Redis failover. Redis had failed over once the previous week and everyone still had that fresh in memory. The Redis primary's metrics were flat, no failover event, no elevated latency on the connections that were already established. It was specifically new connections failing, not existing ones.

That detail was the first real clue, but it took someone cross-referencing the three incident channels to notice it: billing-api, the image-resize worker, and the search proxy shared no code, no deploy history, and no obvious dependency on each other. What they shared was harder to see at first, because none of the three teams had a reason to check it: all three had pods currently scheduled on ip-10-4-21-88.

the hunt

Once the node was the suspect instead of any one service, dmesg on ip-10-4-21-88 gave the answer in one line.

[Mon Aug 24 15:03:41 2026] nf_conntrack: nf_conntrack: table full, dropping packet
[Mon Aug 24 15:03:41 2026] nf_conntrack: nf_conntrack: table full, dropping packet
[Mon Aug 24 15:03:42 2026] nf_conntrack: nf_conntrack: table full, dropping packet

nf_conntrack is the kernel's connection tracking table, every TCP and UDP flow through the node's network stack gets an entry so the kernel knows how to route return traffic, including the NAT hop that every pod-to-service call makes through kube-proxy's iptables rules. It's shared across every pod on the node. Fill it up and the kernel starts dropping new connection attempts node-wide, regardless of which pod or which service is trying to open one.

$ cat /proc/sys/net/netfilter/nf_conntrack_count
65536
$ cat /proc/sys/net/netfilter/nf_conntrack_max
65536

Fully saturated. The next question was what pushed it there. Every pod's /etc/resolv.conf in this cluster carries options ndots:5 and a four-entry search path (default.svc.cluster.local, svc.cluster.local, cluster.local, the node's own VPC domain). A bare name like redis has zero dots, so glibc's resolver tries it against each search suffix in order until one resolves, and it fires the A and AAAA lookups in parallel, so a single successful lookup against the first suffix still costs two DNS queries and two conntrack entries.

conntrack -S during the restart window confirmed it wasn't a fluke reading.

$ conntrack -S | awk '{print $1, $2}'
cpu=0   new=612
cpu=1   new=589
cpu=2   new=417
cpu=3   new=298
# new connections opened on this node in the ~90s restart window, baseline is under 50 per CPU

Just under 1,900 new connections opened on that one node in ninety seconds, almost all of it DNS lookups and PgBouncer connections from notifications-service pods cycling through. The node only had about 1,800 entries of headroom left before the restart started. It didn't take much to tip it over.

the find

Root cause: ip-10-4-21-88 had been running near its nf_conntrack_max ceiling for weeks, a slow accumulation of long-lived keep-alive connections from the general mix of services scheduled on it, with nobody watching that particular metric because it had never crossed the line before. notifications-service's rolling restart didn't do anything wrong on its own, restarting forty pods and reconnecting to three dependencies is completely ordinary. It was just enough new connection and DNS traffic, landing at the wrong moment on an already-saturated node, to tip the table over.

Once it was full, the kernel didn't care which pod owned a given connection attempt. It dropped packets from billing-api reaching Postgres exactly the same way it dropped packets from notifications-service reaching Redis. That's what made the first fifteen minutes so unproductive: three teams were independently debugging their own services for a failure that had nothing to do with any of their code.

the fix

Immediate mitigation was raising the ceiling. A one-line sysctl change on the affected node bought headroom in under a minute, applied cluster-wide with a DaemonSet so it survives node replacement.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: conntrack-tuning
spec:
  selector:
    matchLabels: { app: conntrack-tuning }
  template:
    metadata:
      labels: { app: conntrack-tuning }
    spec:
      hostNetwork: true
      initContainers:
        - name: sysctl
          image: busybox:1.36
          securityContext: { privileged: true }
          command:
            - sh
            - -c
            - |
              sysctl -w net.netfilter.nf_conntrack_max=262144
              sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=3600
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9

The real fix was cutting how many conntrack entries a single rolling restart generates in the first place. Two changes shipped together: a pod-level dnsConfig that puts ndots:1 on notifications-service specifically, since it only ever calls fully-qualified in-cluster service names, and switching the three dependency clients from resolve-per-connection to resolve-once-and-cache on startup.

spec:
  template:
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "1"
      containers:
        - name: notifications-service
          env:
            - name: PGBOUNCER_HOST
              value: pgbouncer.default.svc.cluster.local
            - name: REDIS_HOST
              value: redis.default.svc.cluster.local
            - name: BILLING_API_HOST
              value: billing-api.default.svc.cluster.local

Fully-qualified names with a trailing search-path match skip the multi-suffix search entirely, CoreDNS answers on the first query instead of up to four. Combined with the pool clients resolving once instead of per-connection, the same forty-pod restart today generates under 100 new conntrack entries instead of nearly 1,900.

We also added a Prometheus alert on node_nf_conntrack_entries / node_nf_conntrack_entries_limit crossing 80%, sustained for two minutes. That metric existed already, exported by node-exporter, it just had no alert on it because nothing had ever made it interesting before this.

the aftermath

notifications-service now restarts the same forty pods with about a twentieth of the conntrack footprint it used to. The alert on conntrack utilization hasn't fired since, but it's the first thing anyone checks now when a deploy on one service seems to be breaking a completely different one.