How a gRPC Server's Ping Throttling Policy Let an AWS NLB Silently Kill Connections for 40 Minutes

03:14 UTC. PagerDuty for checkout-svc won't stop. p99 latency on tax calculation calls is at 38 seconds, up from a baseline of 90 milliseconds. Error rate is flat at zero. Nothing is failing. Everything is just... waiting.

the setup

tax-svc runs on a fleet of EC2 instances in an Auto Scaling Group, fronted by an internal AWS Network Load Balancer instead of an ALB, because it's pure gRPC and the team wanted L4 passthrough with no HTTP/2-to-HTTP/1.1 translation in the middle. checkout-svc opens one long-lived gRPC channel to the NLB's DNS name at startup and reuses it for every tax calculation call, roughly 1,800 RPCs a minute at steady state.

Both sides configured gRPC keepalive, which looked like the responsible thing to do. The client pings every 20 seconds to keep the connection warm:

conn, err := grpc.Dial(
  "tax-svc-internal-nlb.example.internal:50051",
  grpc.WithKeepaliveParams(keepalive.ClientParameters{
    Time:                20 * time.Second,
    Timeout:              5 * time.Second,
    PermitWithoutStream: true,
  }),
)

Nobody checked what the server side allowed.

the scramble

First theory: tax-svc is overloaded. CPU, memory, and goroutine counts on every instance are flat and boring. The Auto Scaling Group has three of three healthy targets. Application-level request logs on tax-svc show call volume exactly matching the baseline, no backlog, no slow queries against its local rate tables. Fifteen minutes spent confirming the server is doing nothing wrong, because it genuinely isn't.

Second theory: an AZ-level network event. Three other services call across the same two AZs with no symptoms at all. Pulling up the NLB's CloudWatch metrics, ActiveFlowCount is climbing steadily while NewFlowCount stays flat, which doesn't match a clean network partition. Connections are accumulating somewhere, not failing to form. That rules out the network layer being the actual break and burns another twenty minutes chasing it.

the hunt

With both obvious culprits cleared, the next step is looking at what a single stuck call actually does. A manual gRPC call against tax-svc directly, bypassing the NLB, returns in 40 milliseconds. The exact same call through the NLB, on the channel checkout-svc has been holding since before the incident started, just hangs until the client's own 5-second application deadline kills it. The server never saw the request.

That's the tell: the server isn't slow, the connection is dead, and neither side has noticed. A packet capture on a checkout-svc host, filtered to the NLB's target IP, confirms it. No RST, no FIN. From the kernel's point of view the TCP connection is still fully established. It just stopped carrying anything.

03:09:41 IP checkout-svc.44822 > 10.4.12.9.50051: Flags [P.], length 142
(nothing after this for the life of the connection)

Enabling GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=2 on a checkout-svc instance and restarting it turns up the line that explains everything, logged ninety minutes before the alert fired:

transport: Got too many pings from the client, closing the connection.
http2: received GOAWAY, errcode: 11 (ENHANCE_YOUR_CALM)

the find

tax-svc's gRPC server had its own keepalive enforcement policy, added six weeks earlier to protect against a noisy client that was pinging too aggressively:

grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
  MinTime:             5 * time.Minute,
  PermitWithoutStream: false,
}),

MinTime of five minutes means the server will tolerate a client ping no more often than once every five minutes before deciding the client is abusive and sending GOAWAY. The checkout-svc client was configured to ping every 20 seconds. The first ping inside that five-minute window is fine. The second one, 20 seconds later, gets the server's patience exhausted and a GOAWAY with ENHANCE_YOUR_CALM.

gRPC's client-side keepalive implementation responds to that specific GOAWAY by doubling its ping interval, then doubling it again the next time it gets throttled. Within a handful of cycles the effective ping interval blows past 350 seconds, the NLB's idle timeout for TCP flows. The NLB doesn't know a gRPC ping from application data, it just counts bytes on the wire. Once the real interval crossed 350 seconds, nothing crossed it often enough to reset the idle timer.

When the NLB expires an idle flow, it drops the connection state internally without sending a TCP RST or FIN to either endpoint. Both checkout-svc and tax-svc kept believing the connection was alive. Every RPC checkout-svc sent on that channel went into a TCP send buffer that would never be acknowledged, and sat there until the client's 5-second application deadline finally gave up and returned an error the caller could retry on a fresh connection, except the backed-off keepalive meant the next connection could hit the same throttling and die the same way.

the fix

Immediate mitigation: a rolling restart of checkout-svc to force every channel to re-establish from zero. Latency back to 90ms within four minutes of the first instance restarting. That buys time, it doesn't fix the policy mismatch that caused it.

The real fix is making the server's enforcement policy match a ping interval the client actually needs, with enough margin that one throttling event doesn't cascade into a death spiral of backoffs:

grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
  MinTime:             30 * time.Second,
  PermitWithoutStream: true,
}),
grpc.KeepaliveParams(keepalive.ServerParameters{
  Time:    60 * time.Second,
  Timeout: 10 * time.Second,
}),

checkout-svc's client ping interval was pulled back to 45 seconds, comfortably under the server's new 30-second floor with room to spare, and comfortably under the NLB's 350-second idle timeout even if a ping gets dropped once. As a second, independent layer, the NLB's own idle timeout was raised from the 350-second default to 400 seconds, so a single missed ping cycle no longer has zero margin for error:

resource "aws_lb_target_group" "tax_svc" {
  # ...
  connection_termination = true
  target_health_state {
    unhealthy_draining_interval = 0
  }
}

resource "aws_lb" "tax_svc_nlb" {
  # ...
  load_balancer_type = "network"
  idle_timeout       = 400
}

the aftermath

Both keepalive configs were individually reasonable. The client's 20-second ping was a sane default for a service that cared about fast failure detection. The server's 5-minute enforcement policy was a sane response to a real noisy-neighbor problem from a different client months earlier. Neither side was wrong on its own. They just never agreed to be right together.