How a Bounce-Handling Bug Silently Suppressed Password Reset Emails for 1,800 Users
09:14 UTC, day nine. A support ticket that's been bounced between two tiers finally lands on
an engineer's desk: a customer says she's requested a password reset six times since Monday
and gotten nothing, not even in spam. On-call pulls her account. Email domain:
contractorhq.biz. Two more tickets from the same domain, filed within the hour,
same complaint.
the scramble
First theory: SES is having a bad day. The SES console shows sending quota at 12% utilization and a bounce rate of 0.4%, nowhere near the 5% threshold that gets an account put under review. Not a sending problem.
Second theory: deliverability, SPF or DKIM drifted out of alignment with the receiving domain's filters. Pulling the raw headers from a reset email sent to a different, working address shows both passing clean, DKIM signature valid, SPF aligned. Nothing about the message itself looks like spam.
The pivot comes from checking the app's own send log instead of SES. The password reset
endpoint didn't throw, didn't retry, didn't log an error for any of the three accounts. It
logged email_skipped: suppressed, cleanly, like that was the intended outcome.
Nothing failed. The app chose not to send.
the hunt
Before calling SES for any transactional send, the app checks an internal
email_suppressions table, populated by a Lambda that consumes SES bounce and
complaint notifications off an SNS topic. The three accounts are all in it:
SELECT email, reason, suppressed_at FROM email_suppressions
WHERE email LIKE '%@contractorhq.biz';
email | reason | suppressed_at
-------------------------+-----------+------------------------
j.ruiz@contractorhq.biz | bounce | 2026-10-07 03:12:41+00
t.okafor@contractorhq.biz | bounce | 2026-10-07 03:12:58+00
m.lee@contractorhq.biz | bounce | 2026-10-07 03:13:04+00
Three timestamps, nine days earlier, nineteen seconds apart. That's not three isolated bounces on three separate occasions, that's a batch. Something sent to all three at once and all three bounced the same way within the same window.
The only thing that went out to a list that size that night was a product newsletter, sent to
40,000 subscribers at 03:09 UTC. contractorhq.biz runs its own mail server, and
the timing lines up with a documented, if rare, failure mode on their end: a burst of inbound
mail from one sender hitting a mailbox quota ceiling on several accounts in the same minute.
SES's raw SNS bounce payload for one of the three, pulled from the archived notification, backs
it up:
{
"notificationType": "Bounce",
"bounce": {
"bounceType": "Transient",
"bounceSubType": "General",
"bouncedRecipients": [
{ "emailAddress": "j.ruiz@contractorhq.biz",
"diagnosticCode": "smtp; 452 4.2.2 mailbox full" }
],
"timestamp": "2026-10-07T03:12:41.000Z"
}
}
bounceType: "Transient". A full mailbox, a temporary condition that resolves
itself once the recipient clears space, not a dead address. SES's own documentation is explicit
that Transient bounces should never result in permanent suppression. And yet here
they are, suppressed indefinitely, nine days later.
the find
The bounce-handling Lambda's suppression logic, as deployed:
def handle_bounce(bounce: dict) -> None:
sub_type = bounce["bounceSubType"]
if sub_type == "General":
for r in bounce["bouncedRecipients"]:
suppress_email(r["emailAddress"], reason="bounce")
bounceSubType values aren't unique to one bounceType.
"General" is a valid subtype under both
Permanent (a truly dead address) and Transient (a temporary delivery
problem, mailbox full, message too large, content rejected). The handler branched on the
subtype alone and never checked which top-level type it belonged to, so a wave of ordinary,
self-resolving mailbox-full bounces from one slow mail server got routed through the exact same
path as permanently dead addresses, and suppressed forever on the first occurrence.
The newsletter send was the trigger, not the cause. It was simply the first send in months large enough to hit enough inboxes on one domain, at once, hard enough to push several of them over quota in the same minute. Any password reset or receipt email to those addresses afterward silently skipped SES entirely, logged as a clean, intentional-looking suppression rather than a failure, with no alert tied to it because nothing had technically gone wrong from the app's point of view.
the fix
Immediate containment: identify every suppression entry whose archived bounce payload says
Transient, and release them.
UPDATE email_suppressions s
SET released_at = now()
FROM bounce_events b
WHERE b.email = s.email
AND b.bounce_type = 'Transient'
AND s.released_at IS NULL;
The Lambda now gates on bounceType first, and treats transient bounces as a
backoff signal rather than a ban:
def handle_bounce(bounce: dict) -> None:
bounce_type = bounce["bounceType"]
for r in bounce["bouncedRecipients"]:
email = r["emailAddress"]
if bounce_type == "Permanent":
suppress_email(email, reason="hard_bounce")
elif bounce_type == "Transient":
record_transient_bounce(email)
if transient_bounce_count(email, window_days=7) >= 5:
suppress_email(email, reason="repeated_transient")
A single mailbox-full bounce no longer costs an address its ability to receive a password reset. Five in a week does, which still catches genuinely dead or abandoned inboxes without punishing a recipient for one bad night on their mail server.
The other gap was visibility: email_skipped: suppressed had been treated as a
routine log line, not a signal worth watching. It now feeds a dashboard counter scoped to
password-reset and receipt sends specifically, with an alert if skipped-suppressed count for
either exceeds a handful per hour, since those two email types have no legitimate reason to be
suppressed at any volume.
the aftermath
-
bounceSubTypeis not a reliable signal on its own. The same subtype string can appear under more than onebounceType, and any suppression logic that skips the type check is one coincidence away from treating a temporary problem as a permanent one. - A suppression system needs its own retry and backoff policy for transient failures, not a single binary gate. One bad delivery attempt and a pattern of bad delivery attempts are different signals and deserve different responses.
-
A clean, no-error log line can still be a failure.
skipped: suppressedlooked identical whether it was protecting the system from a dead address or quietly breaking account recovery for a real user. - Transactional email categories that users can't route around, like password resets, need their own monitoring, separate from aggregate sending health. SES's dashboard was green the entire nine days.
Nothing about the send pipeline ever broke. Every password reset email that got suppressed was, from SES's perspective and the app's own logs, working exactly as designed. The design just couldn't tell the difference between an inbox that was full for an hour and one that would never exist again.