How a Swallowed Exit Code Let a Cert Renewal Job Report Success for Three Days Straight

03:14 UTC. PagerDuty: "high error rate: order-svc to fulfillment-svc, 41% of calls failing." No deploy in the last six hours. No infrastructure change flagged in the last day. Whatever broke did it on its own.

the setup

order-svc and fulfillment-svc talk to each other over gRPC, routed through Envoy sidecars inside our Istio mesh. Every call between them is mutually authenticated: each side presents a short-lived certificate signed by an internal CA, and Istio's PeerAuthentication policy rejects any connection where that handshake fails. This pair of services still runs on the original certificate pipeline from before we adopted cert-manager for the rest of the mesh, a bash script in a CronJob that issues a fresh 24-hour certificate every night and pushes it into the two services' Kubernetes secrets.

#!/bin/bash
CA_CERT=/etc/pki/ca/intermediate.pem
CA_KEY=/etc/pki/ca/intermediate.key
CERT=/tmp/fulfillment-svc.crt
KEY=/tmp/fulfillment-svc.key
LOG=/var/log/cert-renew.log

openssl genrsa -out $KEY 2048 | tee -a $LOG
openssl req -new -key $KEY -subj "/CN=fulfillment-svc.mesh" -out /tmp/csr.pem | tee -a $LOG
openssl x509 -req -in /tmp/csr.pem -CA $CA_CERT -CAkey $CA_KEY \
  -CAcreateserial -out $CERT -days 1 | tee -a $LOG

kubectl create secret tls fulfillment-svc-mtls \
  --cert=$CERT --key=$KEY -n mesh --dry-run=client -o yaml | kubectl apply -f -

echo "$(date -u) cert renewed, notAfter: $(openssl x509 -enddate -noout -in $CERT)" >> $LOG

It's the kind of script that gets written once, works, and never gets looked at again. Ours had been running nightly at 20:00 UTC for over a year.

the scramble

On-call pulled up Grafana. The error rate for order-svc's calls into fulfillment-svc had gone from a steady 0.1% to 41% starting at roughly 03:11 UTC, three minutes before the page fired. First theory: a canary rollout for an unrelated service had touched shared Istio VirtualService routing rules two hours earlier. It was the only change anywhere near the mesh config in the last day, and routing changes are a familiar way to break exactly one service pair without touching anything else.

kubectl -n mesh get virtualservice fulfillment-svc -o yaml, diffed against the last known-good version in git: no drift. The rollback would have been a no-op even if applied, so nobody bothered forcing it through.

Second theory: DNS. gRPC connection failures inside a mesh often look like resolution problems before they look like anything else. kubectl exec -it order-svc-7d9f4 -- nslookup fulfillment-svc.mesh.svc.cluster.local resolved cleanly, sub-millisecond, correct cluster IP. CoreDNS wasn't the problem either.

Twenty minutes in, with two plausible theories ruled out and the error rate holding steady at 41%, not climbing and not recovering, someone finally looked at what the actual failing connections said instead of guessing from the outside.

the hunt

Exec'd into an order-svc pod's Envoy sidecar and ran the handshake directly against fulfillment-svc's mesh address.

$ openssl s_client -connect fulfillment-svc.mesh.svc.cluster.local:15006 -showcerts 2>&1 | head -20

CONNECTED(00000003)
...
verify error:num=10:certificate has expired
40802000:error:0A000086:SSL routines:tls_process_server_certificate:certificate verify failed

Expired, not misconfigured, not revoked. Pulling the secret directly and checking its notAfter confirmed it.

$ kubectl -n mesh get secret fulfillment-svc-mtls -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -enddate

notAfter=Aug23 23:58:11 2026 GMT

That certificate had expired at 23:58 UTC the previous night, over three hours before anything actually broke. If it died at 23:58, why did nothing fail until 03:11? gRPC connections between order-svc and fulfillment-svc are long-lived and pooled; the mTLS handshake only happens when a connection is first established, not on every request. The existing pool of open connections kept working straight through the expiry, since nothing forced a fresh handshake. What forced it was a routine autoscaling event around 03:10 that cycled several fulfillment-svc pods, which meant order-svc's connection pool had to open fresh connections to the new pods, and every one of those tried to handshake against a certificate that had already been dead for hours.

That explained the timing. It didn't explain why the nightly renewal hadn't caught it. The CronJob's own logs said otherwise.

2026-08-21 20:00:04 UTC cert renewed, notAfter: Aug22 20:00:03 2026 GMT
2026-08-22 20:00:03 UTC cert renewed, notAfter: Aug22 20:00:03 2026 GMT
2026-08-23 20:00:04 UTC cert renewed, notAfter: Aug22 20:00:03 2026 GMT

Three consecutive nights, three log lines claiming success, and the exact same notAfter timestamp reported all three times. No new certificate had actually been generated since August 21st. The script had been failing silently for three nights and reporting success every time.

the find

Four days earlier, an unrelated project to consolidate three internal certificate authorities into a single directory structure had moved /etc/pki/ca/intermediate.pem to a new path. Every other service using cert-manager's native CA integration picked up the change automatically. This one script, still pointing at the old hardcoded path, did not.

When the script ran that first night after the move, its openssl x509 -req ... -CA $CA_CERT step failed with unable to load certificate: No such file or directory. That should have stopped the script cold. It didn't, because of how the pipeline was written.

openssl x509 -req -in /tmp/csr.pem -CA $CA_CERT -CAkey $CA_KEY \
  -CAcreateserial -out $CERT -days 1 | tee -a $LOG

$? after a pipeline reflects the exit status of the last command in it, tee, not openssl. tee almost never fails. So this line always returned 0, no matter what openssl did upstream, and the script had no set -o pipefail to change that. Because the failing openssl command never overwrote $CERT, the file on disk was still the certificate generated three nights earlier. kubectl apply then re-applied that same, aging certificate into the secret, unchanged, and the script logged a clean success on every one of those three runs.

Root cause: an exit code that mattered got thrown away by a pipe, so a broken renewal job kept reapplying the same expiring certificate for three nights while reporting that it was working. Nothing paged on any of those nights because nothing about a "successful" cron run looked wrong from the outside. It looked exactly like a healthy one.

the fix

Immediate mitigation, to get order-svc and fulfillment-svc talking again without waiting for a proper certificate: flip Istio's PeerAuthentication for that namespace to PERMISSIVE, which accepts both mTLS and plaintext, while a fresh certificate was generated by hand and pushed into the secret manually.

$ kubectl -n mesh patch peerauthentication default \
  --type merge -p '{"spec":{"mtls":{"mode":"PERMISSIVE"}}}'

Real traffic recovered within two minutes of that patch landing. Fixing the actual pipeline took longer, and had three parts. First, the script fails loudly now instead of quietly:

#!/bin/bash
set -euo pipefail

CA_CERT=/etc/pki/ca/current/intermediate.pem
CA_KEY=/etc/pki/ca/current/intermediate.key
CERT=/tmp/fulfillment-svc.crt
KEY=/tmp/fulfillment-svc.key
LOG=/var/log/cert-renew.log

prev_not_after=$(openssl x509 -enddate -noout -in "$CERT" 2>/dev/null | cut -d= -f2 || echo "none")

openssl genrsa -out "$KEY" 2048
openssl req -new -key "$KEY" -subj "/CN=fulfillment-svc.mesh" -out /tmp/csr.pem
openssl x509 -req -in /tmp/csr.pem -CA "$CA_CERT" -CAkey "$CA_KEY" \
  -CAcreateserial -out "$CERT" -days 1

new_not_after=$(openssl x509 -enddate -noout -in "$CERT" | cut -d= -f2)
if [ "$new_not_after" = "$prev_not_after" ]; then
  echo "$(date -u) FATAL: renewed cert has same notAfter as previous, refusing to apply" >> "$LOG"
  exit 1
fi

kubectl create secret tls fulfillment-svc-mtls \
  --cert="$CERT" --key="$KEY" -n mesh --dry-run=client -o yaml | kubectl apply -f -
echo "$(date -u) cert renewed, notAfter: $new_not_after" >> "$LOG"

set -euo pipefail means any failing command anywhere in the script now stops it and returns a non-zero exit code, and the CronJob's own failure gets surfaced through Kubernetes as a failed job instead of a quiet success. The explicit notAfter comparison is the second line of defense: even if a future failure somehow still exits clean, reapplying an unchanged certificate now fails the run on purpose.

Second, every CronJob script in this repo now runs through shellcheck in CI, with SC2086 and pipefail-related warnings treated as errors rather than suggestions.

Third, a synthetic probe now runs every five minutes from a canary pod, opening a fresh mTLS connection to each service pair in the mesh and pushing the resulting certificate's remaining lifetime as a metric. An alert fires at 48 hours before expiry, well before any renewal job's own self-reported success could hide a real failure again.

the aftermath

No orders were lost and no customer was charged incorrectly. order-svc's own retry queue held every failed fulfillment reservation and replayed them once mTLS came back, though 2,100 orders had their shipment record created a median of 41 minutes late, and a handful of same-day delivery orders missed their cutoff window entirely.

The old script still exists in git history as a reminder. Every CronJob touching a certificate now gets its exit code checked by something other than itself.