October 26, 2026Architecture8 min read

How One 2.8-Million-Member Sorted Set Froze Redis Mid-Reshard, Two Days Before a Flash Sale

Published October 26, 20268 min read

11:42 UTC, two days before the sale goes live. Datadog pages on-call: redis_p99_latency over 2000ms on three of six shards. Nobody's running load tests. There's no traffic to speak of yet. The only thing happening on the cluster right now is the resharding job that's supposed to make this weekend boring.

the scramble

The team had provisioned three new primaries the day before, planning to migrate half the cluster's 16,384 hash slots onto them with redis-cli --cluster reshard, finish early, and have a full day to watch it sit quietly before the sale started. The reshard kicked off at 11:30 UTC. Twelve minutes later, the pages started.

First theory: the new nodes are undersized. Someone pulls up top on the affected shard. 21% CPU, memory nowhere near the configured maxmemory, disk iowait at zero. The box looks bored, not overloaded, which makes the latency graph look like a lie.

Second theory: the new primaries sit in a second availability zone, added for redundancy as part of this scale-up, and cross-AZ latency is the usual suspect when a topology change coincides with a slowdown. Ping times between the zones come back at 0.6ms. That's not six seconds of anything.

Third theory: a noisy neighbor on shared EC2 hardware. CloudWatch shows no CPU steal, no burst balance depletion, nothing unusual on the instance. Three plausible causes, three dead ends, and the latency graph keeps climbing in the same sawtooth pattern every few minutes.

the hunt

The reshard job is still running in a tmux session from the first engineer who kicked it off. Someone scrolls up through its output and lines the timestamps up against the Datadog graph on a second monitor. Every latency spike lands within a second of a "Moving slot" line in the migration log. Not traffic-correlated. Migration-correlated.

redis-cli --cluster reshard output

Moving slot 7823 from 10.0.4.12:6379 to 10.0.6.21:6379:
Moving slot 7824 from 10.0.4.12:6379 to 10.0.6.21:6379: .
Moving slot 7825 from 10.0.4.12:6379 to 10.0.6.21:6379: ....
Moving slot 7826 from 10.0.4.12:6379 to 10.0.6.21:6379: .......
  (6.1s pause between dots)

That last slot stalls for over six seconds mid-migration, no output, then resumes. INFO commandstats on the sending node confirms where the time is going.

redis-cli -h 10.0.4.12 info commandstats | grep migrate

cmdstat_migrate:calls=412,usec=38204991,usec_per_call=92731.53,
  rejected_calls=0,failed_calls=0

Ninety-two milliseconds average is already odd for a command that's supposed to move one key at a time in small batches. CLIENT LIST on the same node shows the real picture: dozens of client connections sitting with high age and no movement, all queued behind something that isn't releasing the main thread. Pausing the reshard and running a scan on the slot it stalled on answers the question.

redis-cli --bigkeys (filtered to the stalled slot's keyspace)

Biggest sorted set found 'trending:global:zscore' has 2834107 members
Biggest sorted set size in bytes is approximately 341 MB

trending:global:zscore backs the homepage "trending now" rail. Every product view across the entire catalog calls ZINCRBY against it. It has no TTL, no cap, and no trim job. Fourteen months of launch traffic had grown it to 2.8 million members with nobody noticing, because a sorted set that size behaves fine under normal reads and writes. It only becomes a problem the moment something tries to move the whole thing at once.

the find

MIGRATE is not a background operation. It serializes the entire value into a DUMP payload on the source node and streams it to the destination, which issues RESTORE and acknowledges. Redis processes commands on a single thread, so for the entire time it takes to pack and transmit that payload, the node can't service anything else. Not just the key being migrated. Every client on that shard, including ones asking about session lookups or cart contents that have nothing to do with trending products.

The math lines up with the stall almost exactly. Measured throughput on that intra-VPC link for a payload this size came out to roughly 53MB/s once serialization overhead is included. 341MB divided by 53MB/s is about 6.4 seconds, the same length as the pause in the reshard log and the same width as the latency spikes on the dashboard. The "21% CPU" that looked like proof the node was fine was an aggregate across eight cores. The one core running Redis's command loop was pegged the entire time; the other seven were idle and dragged the average down enough to hide it.

the fix

The immediate move was to abort the in-flight reshard and exclude the slot holding trending:global:zscore until the key itself was dealt with. The real fix was making sure no single key could ever grow large enough to block a migration again.

The counter was split by day, using a hash tag so the per-day keys still colocate on one shard for atomic scatter-gather reads, and each day's set gets trimmed to the members that actually matter for ranking.

app/trending.py

def record_view(product_id: str) -> None:
    day = datetime.utcnow().strftime("%Y%m%d")
    key = f"trending:{day}:zscore"
    redis.zincrby(key, 1, product_id)
    redis.expire(key, 60 * 60 * 24 * 3)  # 3-day rolling window

def top_trending(limit: int = 50) -> list[str]:
    today = datetime.utcnow().strftime("%Y%m%d")
    key = f"trending:{today}:zscore"
    redis.zremrangebyrank(key, 0, -50_001)  # cap growth, keep top 50k
    return redis.zrevrange(key, 0, limit - 1)

Each day's key now self-limits to 50,000 members instead of growing without bound, and a 3-day TTL means stale days fall out of the keyspace on their own. The largest a single key can get is bounded by design, not by hoping nobody forgets to run a cleanup job.

Second, the reshard runbook got a mandatory gate before it's allowed to start.

scripts/pre-reshard-check.sh

#!/usr/bin/env bash
set -euo pipefail
MAX_BYTES=10000000  # 10MB per key

redis-cli --bigkeys -h "$1" 2>&1 | tee /tmp/bigkeys.out
if grep -qE "size in bytes is approximately ([0-9]{8,})" /tmp/bigkeys.out; then
  echo "BLOCKED: a key over the size threshold exists. Resolve before resharding." >&2
  exit 1
fi
echo "OK: no oversized keys found, safe to reshard."

Third, a scheduled key-size scan now runs independently of any migration, so an oversized key gets flagged on a Tuesday afternoon instead of during the next scale-up.

the aftermath

341MB Size of the single sorted set that blocked MIGRATE on its source shard

6.4s Blocking duration per migrated big-key transfer, measured and matched to the throttled link speed

3 of 6 Shards affected, each time a slot holding part of the oversized key's data moved

21% Aggregate CPU reported during the stall, which hid a single core pegged at 100%

MIGRATE is not incremental for a single key. It serializes and ships the whole value in one blocking operation, no matter how small the --cluster-pipeline batch size is set for the keys around it.
Aggregate CPU percentage across cores can hide single-thread saturation completely. A Redis node pegged on its one command-processing core can still report a bored-looking total.
A key with no cap and no TTL that grows slowly under normal traffic for over a year can still become the exact thing that breaks an infrastructure change, because normal reads and writes never exercise the operation that moves the whole value at once.
Resharding checklists need a key-size gate, not just a slot-count estimate. The slot distribution looked perfectly balanced; the byte distribution inside one slot didn't.

Nobody had written a bad line of code to get here. ZINCRBY on a global counter is a reasonable way to build a trending rail, and it worked for fourteen months without a single complaint. It just never had to move. The first time it did, it took three shards down with it, two days before the one event this scale-up was supposed to protect.