How a Three-Value Partition Key Turned One Kinesis Shard Into a Bottleneck and Dropped 38% of Events

14:32 UTC, Tuesday. A Datadog monitor fires: ingest rate for web events is down 38% against the 7-day baseline. Not a spike, a drop. No deploy in the last six hours to the consumer side, no error budget burn anywhere, and the on-call engineer's first read of the dashboards is that nothing is actually broken, just quieter than usual.

the scramble

First theory: the downstream Lambda that reads from the stream and feeds the anomaly detection pipeline has started failing silently. The Lambda's error rate is flat at zero, and invocation count is down by almost exactly the same percentage as the event drop, which at first reads as consistent with a consumer problem, until someone points out that a consumer reading less data because less data arrived looks identical to a consumer that's broken. That's dead end number one, and it costs fifteen minutes.

Second theory: the producer service itself is unhealthy. Its health checks are green, CPU and memory are unremarkable, and there are no 5xx responses in its own access logs. The producer calls PutRecords against the stream on a steady schedule and logs a warning on any throttling response, with automatic retry and backoff built into the AWS SDK. A quick grep of those logs for the last hour turns up something, but it looks survivable:

412

412 throttled calls in an hour sounds like exactly the kind of transient backpressure the SDK's retry logic exists to absorb, and every one of those 412 log lines is followed a few hundred milliseconds later by a successful retry. Nothing in the producer logs says "events are being lost." That's dead end number two: the retries really were mostly succeeding, which hid the fact that something underneath them was badly out of balance.

the hunt

The stream-level CloudWatch metrics are the next stop, and they're where the picture gets confusing in a new way:

Sum: ~5% below the 7-day baseline

Five percent, not thirty-eight. The stream's own ingestion metric barely moved, while the dashboard tracking web-source events specifically showed a 38% drop. Both numbers are real, which means the 33-point gap between them has to live somewhere CloudWatch's stream-level view can't show: inside the distribution of traffic across the stream's 16 shards. And that's the problem, because shard-level metrics on Kinesis are opt-in and nobody had turned them on, since they cost extra and the stream had never needed them before.

{
    "StreamName": "events-stream",
    "CurrentShardLevelMetrics": [],
    "DesiredShardLevelMetrics": [
        "IncomingRecords",
        "WriteProvisionedThroughputExceeded"
    ]
}

It takes about five minutes for the new per-shard data to start showing up. When it does, one shard ID accounts for effectively all of the throttling:

shardId-000000000003   412
shardId-000000000000     0
shardId-000000000001     0
shardId-000000000002     0
shardId-000000000004     0
... (11 more shards)      0

One shard is pinned against its 1,000-records-per-second write limit while the other fifteen sit nearly idle. The stream's combined capacity is nowhere near saturated, which is exactly why the aggregate metric looked almost normal, it was averaging one maxed-out shard against fifteen quiet ones.

git log --since="9am" -- producer/ turns up the actual cause fast: a deploy that morning at 08:10 UTC changed the field used as the Kinesis PartitionKey.

client.put_records(
    StreamName=STREAM_NAME,
    Records=[{
        'Data': json.dumps(event).encode(),
        'PartitionKey': event['user_id'],
    } for event in batch],
)

client.put_records(
    StreamName=STREAM_NAME,
    Records=[{
        'Data': json.dumps(event).encode(),
        'PartitionKey': event['source'],  # 'web' | 'mobile' | 'batch'
    } for event in batch],
)

The intent behind the change was reasonable on its face: a downstream fan-out job wanted events grouped by source, and keying by source looked like a cheap way to get that grouping for free from Kinesis's own hashing. But Kinesis assigns a record to a shard by MD5-hashing the partition key into a 128-bit value and mapping it into a shard's hash range. event['source'] only ever takes one of three values, and roughly 95% of all events are 'web'. MD5 of the literal string 'web' is a fixed value, every single time, so every web event landed on whichever one shard's hash range happened to contain it. Three possible keys collapsed sixteen shards' worth of parallelism down to effectively one.

the find

The partition key refactor concentrated 95% of the stream's traffic onto a single shard's 1,000-record-per-second, 1 MiB-per-second limit, while the stream's other fifteen shards carried almost nothing. The aggregate CloudWatch metrics that everyone actually watches average across all sixteen shards, so a single shard pinned at 100% utilization barely moves the stream-wide number. Shard-level metrics would have shown it instantly, but those are opt-in and had never been turned on, because the stream had never had a reason to need them, until a three-value key made that reason arrive without warning.

the fix

The first instinct was to split the hot shard, on the theory that more shards would spread the load:

{}

Five minutes later, shard-level metrics showed the exact same throttling pattern, just relocated to a different single shard ID. Splitting didn't help, and it couldn't have: MD5 of the string 'web' is one fixed point in the hash space, and a split only divides a shard's hash range into two sub-ranges. Whichever side of the split contains that one fixed point inherits all of the traffic. Resharding fixes a hot range of many distinct keys. It does nothing for a hot point created by a low-cardinality key, because there's no way to split a single fixed hash value in half.

The real fix was reverting the partition key to something with enough cardinality to actually spread across shards, and recovering the source grouping a different way: compound it into the key instead of using it alone, so cardinality comes from user_id and the source tag just rides along for the downstream consumer to filter on:

client.put_records(
    StreamName=STREAM_NAME,
    Records=[{
        'Data': json.dumps(event).encode(),
        'PartitionKey': f"{event['source']}:{event['user_id']}",
    } for event in batch],
)

Shards exist to scale raw throughput, not to pre-group data by category. The downstream fan-out job that wanted per-source grouping now gets it by filtering on the source field in the payload after reading, which costs nothing compared to a stream that can't absorb its own traffic.

the aftermath

Most of the delayed records eventually went through once retries succeeded, but not all of them. Each producer host kept a bounded in-memory retry queue capped at 10,000 records to avoid unbounded memory growth during exactly this kind of sustained throttling, and across the six-host producer fleet, that cap was hit roughly six minutes into the incident. Records that arrived after the queue was full were dropped at the application layer with a log line nobody was alerting on, which is where the true, unrecoverable 38% figure comes from. The retry mechanism that looked reassuring in the first ten minutes was quietly running out of room the whole time.

Sixteen shards of aggregate capacity meant nothing once three possible values for one field decided which single shard actually did the work. The stream was never overloaded. One sixteenth of it was.