How a Bounce-Handling Bug Silently Suppressed Password Reset Emails for 1,800 Users
← Back
October 16, 2026Architecture7 min read

How a Bounce-Handling Bug Silently Suppressed Password Reset Emails for 1,800 Users

Published October 16, 20267 min read

09:14 UTC, day nine. A support ticket that's been bounced between two tiers finally lands on an engineer's desk: a customer says she's requested a password reset six times since Monday and gotten nothing, not even in spam. On-call pulls her account. Email domain: contractorhq.biz. Two more tickets from the same domain, filed within the hour, same complaint.


the scramble

First theory: SES is having a bad day. The SES console shows sending quota at 12% utilization and a bounce rate of 0.4%, nowhere near the 5% threshold that gets an account put under review. Not a sending problem.

Second theory: deliverability, SPF or DKIM drifted out of alignment with the receiving domain's filters. Pulling the raw headers from a reset email sent to a different, working address shows both passing clean, DKIM signature valid, SPF aligned. Nothing about the message itself looks like spam.

The pivot comes from checking the app's own send log instead of SES. The password reset endpoint didn't throw, didn't retry, didn't log an error for any of the three accounts. It logged email_skipped: suppressed, cleanly, like that was the intended outcome. Nothing failed. The app chose not to send.


the hunt

Before calling SES for any transactional send, the app checks an internal email_suppressions table, populated by a Lambda that consumes SES bounce and complaint notifications off an SNS topic. The three accounts are all in it:

email_suppressions, query against the three accounts
SELECT email, reason, suppressed_at FROM email_suppressions
WHERE email LIKE '%@contractorhq.biz';

         email          |  reason   |     suppressed_at
-------------------------+-----------+------------------------
 j.ruiz@contractorhq.biz | bounce    | 2026-10-07 03:12:41+00
 t.okafor@contractorhq.biz | bounce  | 2026-10-07 03:12:58+00
 m.lee@contractorhq.biz  | bounce    | 2026-10-07 03:13:04+00

Three timestamps, nine days earlier, nineteen seconds apart. That's not three isolated bounces on three separate occasions, that's a batch. Something sent to all three at once and all three bounced the same way within the same window.

The only thing that went out to a list that size that night was a product newsletter, sent to 40,000 subscribers at 03:09 UTC. contractorhq.biz runs its own mail server, and the timing lines up with a documented, if rare, failure mode on their end: a burst of inbound mail from one sender hitting a mailbox quota ceiling on several accounts in the same minute. SES's raw SNS bounce payload for one of the three, pulled from the archived notification, backs it up:

SES bounce notification, archived SNS message body
{
  "notificationType": "Bounce",
  "bounce": {
    "bounceType": "Transient",
    "bounceSubType": "General",
    "bouncedRecipients": [
      { "emailAddress": "j.ruiz@contractorhq.biz",
        "diagnosticCode": "smtp; 452 4.2.2 mailbox full" }
    ],
    "timestamp": "2026-10-07T03:12:41.000Z"
  }
}

bounceType: "Transient". A full mailbox, a temporary condition that resolves itself once the recipient clears space, not a dead address. SES's own documentation is explicit that Transient bounces should never result in permanent suppression. And yet here they are, suppressed indefinitely, nine days later.


the find

The bounce-handling Lambda's suppression logic, as deployed:

bounce-handler/index.py, as found
def handle_bounce(bounce: dict) -> None:
    sub_type = bounce["bounceSubType"]
    if sub_type == "General":
        for r in bounce["bouncedRecipients"]:
            suppress_email(r["emailAddress"], reason="bounce")

bounceSubType values aren't unique to one bounceType. "General" is a valid subtype under both Permanent (a truly dead address) and Transient (a temporary delivery problem, mailbox full, message too large, content rejected). The handler branched on the subtype alone and never checked which top-level type it belonged to, so a wave of ordinary, self-resolving mailbox-full bounces from one slow mail server got routed through the exact same path as permanently dead addresses, and suppressed forever on the first occurrence.

The newsletter send was the trigger, not the cause. It was simply the first send in months large enough to hit enough inboxes on one domain, at once, hard enough to push several of them over quota in the same minute. Any password reset or receipt email to those addresses afterward silently skipped SES entirely, logged as a clean, intentional-looking suppression rather than a failure, with no alert tied to it because nothing had technically gone wrong from the app's point of view.


the fix

Immediate containment: identify every suppression entry whose archived bounce payload says Transient, and release them.

one-off reconciliation script
UPDATE email_suppressions s
SET released_at = now()
FROM bounce_events b
WHERE b.email = s.email
  AND b.bounce_type = 'Transient'
  AND s.released_at IS NULL;

The Lambda now gates on bounceType first, and treats transient bounces as a backoff signal rather than a ban:

bounce-handler/index.py, after
def handle_bounce(bounce: dict) -> None:
    bounce_type = bounce["bounceType"]
    for r in bounce["bouncedRecipients"]:
        email = r["emailAddress"]
        if bounce_type == "Permanent":
            suppress_email(email, reason="hard_bounce")
        elif bounce_type == "Transient":
            record_transient_bounce(email)
            if transient_bounce_count(email, window_days=7) >= 5:
                suppress_email(email, reason="repeated_transient")

A single mailbox-full bounce no longer costs an address its ability to receive a password reset. Five in a week does, which still catches genuinely dead or abandoned inboxes without punishing a recipient for one bad night on their mail server.

The other gap was visibility: email_skipped: suppressed had been treated as a routine log line, not a signal worth watching. It now feeds a dashboard counter scoped to password-reset and receipt sends specifically, with an alert if skipped-suppressed count for either exceeds a handful per hour, since those two email types have no legitimate reason to be suppressed at any volume.


the aftermath

1,800 Addresses incorrectly suppressed from one newsletter send
9 days From the bad bounce wave to the first escalated ticket
212 Password reset and receipt sends silently skipped in that window
0 Errors logged for any of the skipped sends
  • bounceSubType is not a reliable signal on its own. The same subtype string can appear under more than one bounceType, and any suppression logic that skips the type check is one coincidence away from treating a temporary problem as a permanent one.
  • A suppression system needs its own retry and backoff policy for transient failures, not a single binary gate. One bad delivery attempt and a pattern of bad delivery attempts are different signals and deserve different responses.
  • A clean, no-error log line can still be a failure. skipped: suppressed looked identical whether it was protecting the system from a dead address or quietly breaking account recovery for a real user.
  • Transactional email categories that users can't route around, like password resets, need their own monitoring, separate from aggregate sending health. SES's dashboard was green the entire nine days.

Nothing about the send pipeline ever broke. Every password reset email that got suppressed was, from SES's perspective and the app's own logs, working exactly as designed. The design just couldn't tell the difference between an inbox that was full for an hour and one that would never exist again.

Share this
← All Posts7 min read