How the DST Fall-Back Hour Made Our Rate Limiter Forget the Last Minute, Every Minute, for an Hour

01:47 EST, Sunday, November 1st. PagerDuty: "checkout-api: p99 latency 4.2s, up from 180ms." Two minutes later, a second alert: "inventory-svc: request rate 2.3x baseline." Nothing had deployed in six hours. Nothing had changed in the traffic mix. The dashboards just started climbing at 01:00 and kept going.

the scramble

First theory, the obvious one for a Sunday night: bot traffic. On-call pulled the Cloudflare firewall events for the last hour. Nothing unusual, no new ASNs, no spike in a single IP range, no user-agent anomaly. Whatever was hitting checkout was behaving like normal browser traffic, just more of it than the rate limiter should have allowed through.

Second theory: the rate limiter itself was down and failing open. It wasn't. The limiter's own health check was green, and its own admitted-request counter, the one it logs internally, still showed traffic under the configured cap of 600 requests per minute per client. That was the detail that mattered and got skipped past twice before anyone noticed it: the limiter believed it was doing its job. The actual request volume hitting inventory-svc downstream said otherwise.

Third theory came from someone who'd been stress-testing checkout the week before and still had a terminal open on the limiter's Redis instance. Go look at what the buckets actually contain.

> KEYS ratelimit:client_8841:*
ratelimit:client_8841:202611010147
> TTL ratelimit:client_8841:202611010147
(integer) 43
> GET ratelimit:client_8841:202611010146
(nil)

One bucket for the current minute, nothing for the one right before it. On a normal night that nil would be unremarkable, buckets expire, that's the point. But the sliding window logic summed the current minute's count with the previous minute's count to decide whether a request was over the limit, and if the previous bucket kept coming back empty, the limiter was only ever seeing one minute of traffic at a time. Effectively half its memory, every single minute, for the whole hour.

the hunt

The bucket key was the first thing worth reading closely. 202611010147 wasn't a UTC timestamp, it was a formatted local datetime: YYYYMMDDHHMM in America/New_York. That had been true since the limiter shipped eight months earlier, written by someone who wanted human-readable Redis keys for on-call debugging and didn't think about what "local" meant for arithmetic.

from datetime import datetime, timedelta
from zoneinfo import ZoneInfo

LOCAL_TZ = ZoneInfo("America/New_York")
BUCKET_TTL_SECONDS = 120
LIMIT_PER_MINUTE = 600

def bucket_key(client_id: str, when: datetime) -> str:
    local = when.astimezone(LOCAL_TZ)
    return f"ratelimit:{client_id}:{local.strftime('%Y%m%d%H%M')}"

def is_allowed(client_id: str) -> bool:
    now = datetime.now(LOCAL_TZ)
    current_key = bucket_key(client_id, now)
    previous_key = bucket_key(client_id, now - timedelta(minutes=1))

    current = int(redis.get(current_key) or 0)
    previous = int(redis.get(previous_key) or 0)

    # weight the previous bucket by how far into the current minute we are
    elapsed_fraction = now.second / 60
    weighted = current + previous * (1 - elapsed_fraction)

    if weighted >= LIMIT_PER_MINUTE:
        return False

    pipe = redis.pipeline()
    pipe.incr(current_key)
    pipe.expire(current_key, BUCKET_TTL_SECONDS)
    pipe.execute()
    return True

This is a standard sliding-window-counter implementation, the kind that shows up in most rate limiting writeups, ours included. It works because now - timedelta(minutes=1) should point at the bucket key that was written sixty real seconds ago. On November 1st at 2:00 AM local, clocks in America/New_York fell back to 1:00 AM, and the hour from 1:00 to 2:00 happened twice. During the second pass through it, now - timedelta(minutes=1), computed on an aware datetime, correctly subtracted sixty real seconds, that part of Python's datetime arithmetic is timezone-correct. The break was one line earlier: bucket_key() immediately threw that correctness away by formatting the result back down to a naive string with no offset in it. 01:47 EDT and 01:47 EST, sixty minutes apart in real time, format to the exact same key: 202611010147.

That collision wasn't the direct problem, the bucket from the first pass through 1:47 AM had long since expired (120 second TTL) by the time the clock looped back to it an hour later. The direct problem was what the collision did to the previous-minute lookup. During the repeated hour, now was local time that had already been used once before, an hour earlier in real time. Subtracting one minute from it produced a previous-key string that had also already been used and expired an hour ago, well outside its 120-second TTL. Every single lookup for the previous bucket came back nil, for the entire repeated hour, not because of a collision but because the naive key format made the service ask Redis for a bucket that, from Redis's perspective, was ancient history.

With previous pinned at zero, the weighted sum in is_allowed() reduced to just the current minute's count. The limiter wasn't blocking anything until a single minute bucket alone hit 600, instead of the rolling 600-per-any-sixty-seconds it was designed to enforce. A client sending 550 requests in the last five seconds of one minute and another 550 in the first five seconds of the next sailed through both checks, because neither minute's bucket crossed the threshold on its own. Sustained throughput roughly doubled for the hour, and it landed on a Sunday night when nothing else was different enough to make it obvious sooner.

the find

Root cause: the rate limiter's sliding window depended on the previous minute's Redis key being reachable by subtracting one minute and reformatting. Formatting that lookup key from local wall clock time instead of UTC meant the fall-back DST transition, where local time is not monotonic, produced previous-minute keys that pointed at buckets from an hour earlier rather than sixty seconds earlier. Those buckets had long since expired, so the previous-minute contribution silently dropped to zero for the entire repeated hour, and the limiter enforced roughly half its configured strictness without any error, alert, or log line indicating degraded behavior. It looked, from every internal metric the limiter exposed, like it was working correctly.

the fix

The fix was to stop using local time in the bucket key entirely. UTC doesn't observe DST, so a UTC-epoch-derived key is monotonic across every real minute boundary, including the ones that happen twice or get skipped on local clocks:

import time

BUCKET_TTL_SECONDS = 120
LIMIT_PER_MINUTE = 600

def bucket_key(client_id: str, epoch_seconds: float) -> str:
    minute_bucket = int(epoch_seconds // 60)
    return f"ratelimit:{client_id}:{minute_bucket}"

def is_allowed(client_id: str) -> bool:
    now = time.time()  # UTC epoch seconds, unaffected by local DST rules
    current_key = bucket_key(client_id, now)
    previous_key = bucket_key(client_id, now - 60)

    current = int(redis.get(current_key) or 0)
    previous = int(redis.get(previous_key) or 0)

    elapsed_fraction = (now % 60) / 60
    weighted = current + previous * (1 - elapsed_fraction)

    if weighted >= LIMIT_PER_MINUTE:
        return False

    pipe = redis.pipeline()
    pipe.incr(current_key)
    pipe.expire(current_key, BUCKET_TTL_SECONDS)
    pipe.execute()
    return True

The local-time formatting is still useful for on-call debugging, so we kept it, but as a separate, non-functional label attached to the same Redis key rather than as the key itself:

> HGETALL ratelimit:client_8841:29283412
count            412
local_label      2026-11-01 01:47 EST

We also added a test that fast-forwards the process clock across a synthetic DST fall-back boundary and asserts the previous-minute bucket is still reachable, so this class of bug fails in CI instead of on the one Sunday night a year it can actually happen:

def test_previous_bucket_reachable_across_dst_fallback(fake_redis, freeze_time):
    # 2026-11-01 01:47 EDT, then again 2026-11-01 01:47 EST, one real hour later
    freeze_time.move_to("2026-11-01T05:47:00+00:00")  # 01:47 EDT
    is_allowed("client_test")
    freeze_time.move_to("2026-11-01T06:47:00+00:00")  # 01:47 EST, same local clock string
    key_now = bucket_key("client_test", time.time())
    key_60s_ago = bucket_key("client_test", time.time() - 60)
    assert key_now != key_60s_ago

Last, an alert directly on the signal that would have surfaced this in minutes instead of the 47 it actually took: a comparison between the limiter's admitted-request count and the actual request volume the downstream service recorded.

metric: ratio(inventory_svc.requests_received, ratelimit.requests_admitted)
scope: service:checkout-api
threshold: > 1.15 for 10 minutes
message: "Downstream request volume exceeds what the rate limiter reports admitting.
  Check whether the limiter's window logic is degraded before assuming a traffic spike."

the aftermath

Inventory-svc held, barely, its own connection pool queued instead of rejecting, which is the only reason this stayed a latency incident instead of a checkout outage. Nothing was overwritten, no orders were lost, no data was wrong. The cost was entirely in degraded service during a window when nobody could explain why a system that claimed to be enforcing its limit clearly wasn't.

The fix shipped Monday morning, five lines changed. The bug had been live for eight months, correct every single day except the one night a year local time folds back on itself, which is exactly the kind of bug that survives code review, survives load testing, and only introduces itself once, on a clock nobody thought to distrust.