How Dropping a Postgres server_default During an ORM Refactor Let a Bulk Importer Silently Zero Out 214 Invoices

06:42 UTC. A Datadog monitor that has never fired once in the eight months since it was set up goes red: billing.nightly_invoiced_revenue_usd is 18% below its 7-day trailing average. The on-call billing engineer's first instinct is that the threshold is just noisy, revenue monitors usually are, until she opens the dashboard and sees the drop sitting flat across the whole night's batch run, not one spiky data point. Finance hasn't noticed yet. The invoicing job itself reports success: every subscription due last night got an invoice row. The money just isn't on them.

the setup

subscriptions.billing_cycle_days had been a simple column for two years: INTEGER NOT NULL DEFAULT 30. Every plan billed monthly, so a fixed default at the database level was correct for every write path, no matter which one it came from.

That stopped being true when product shipped a "flex" tier with a 7-day cycle and a "custom" enterprise tier with a cycle set per contract. A fixed default of 30 was now wrong for two of three plans, so the fix moved the default into the application layer, where it could look at the plan being assigned and compute the right number:

def default_cycle_days(context):
    plan_id = context.get_current_parameters()['plan_id']
    return PLAN_CYCLE_DAYS.get(plan_id, 30)

class Subscription(Base):
    __tablename__ = 'subscriptions'
    billing_cycle_days = Column(Integer, nullable=True, default=default_cycle_days)

The Alembic autogenerate diff for this change was clean and, on its own, correct:

def upgrade():
    op.alter_column(
        'subscriptions', 'billing_cycle_days',
        server_default=None,
        existing_type=sa.Integer(),
    )

Removing a server-side default that the application now computes itself is a reasonable cleanup. The column had been left nullable since the original rollout, for a backward-compatibility window that was supposed to be temporary and never got revisited, so nothing in the schema would stop a NULL from landing there. Every write path that went through Subscription(...) got the new logic. The migration shipped on a Tuesday, passed review, and nothing broke that week.

the scramble

First theory: the win-back discount campaign running that month. A promo code bug stacking discounts could easily shave money off invoices. Coupon redemption counts for the night are completely normal, same ballpark as the last three weeks. Dead end inside two minutes.

Second theory: a Stripe-side issue silently discounting or voiding charges. Stripe's status page is green, webhook success rate for the night is 100%, and the invoice objects created in Stripe match the invoice rows created locally one for one. Dead end.

The actual lead comes from re-reading the invoicing job's own success log more carefully: the count of invoices generated last night is normal, 1,842, right in line with a typical Tuesday. The total dollar amount is down 18%. That's not a job failing to run. That's a job running fine and generating invoices with the wrong amount on them.

the hunt

Grouping last night's invoices by amount turns up the shape of the problem immediately:

SELECT amount_due, count(*) FROM invoices
WHERE created_at >= now() - interval '18 hours'
GROUP BY amount_due ORDER BY 2 DESC LIMIT 5;

 amount_due | count
------------+-------
      29.00 |   893
      99.00 |   611
       0.00 |   214
      49.00 |   124

214 invoices at exactly $0.00 overnight, against a trailing average of roughly four. Joining those back to subscriptions finds the common thread:

SELECT s.id, s.plan_id, s.billing_cycle_days, s.created_at
FROM invoices i JOIN subscriptions s ON s.id = i.subscription_id
WHERE i.amount_due = 0 AND i.created_at >= now() - interval '18 hours'
LIMIT 5;

 id   | plan_id | billing_cycle_days | created_at
------+---------+---------------------+---------------------
 9981 | flex    |       [NULL]        | 2026-09-24 02:11:03
 9982 | flex    |       [NULL]        | 2026-09-24 02:11:03
 9984 | custom  |       [NULL]        | 2026-09-24 02:11:04

Every one of the 214 has billing_cycle_days IS NULL, and every one was created on or after September 24th, eleven days earlier. alembic history confirms the server_default migration landed on September 24th too, at 09:00 UTC, hours before that night's first bad row. That's a correlation worth chasing but not yet an explanation, since the ORM default should have filled in a real number on every insert after that migration.

The actual insert path for these 214 rows isn't the ORM at all. They came from the nightly win-back reactivation job, a standalone script that re-enables churned accounts who respond to a retention email by bulk-loading them straight into Postgres:

copy_sql = """
    COPY subscriptions (id, user_id, plan_id, status, next_invoice_date)
    FROM STDIN WITH (FORMAT csv)
"""
with conn.cursor() as cur, cur.copy(copy_sql) as copy:
    for row in cohort_rows:
        copy.write_row(row)

billing_cycle_days was never in that column list, before or after the refactor. Before September 24th that didn't matter, because Postgres filled in 30 for any column with a server-side default that an INSERT or COPY left out, regardless of which code wrote the row. After September 24th, there was no server-side default left to fall back on, the column was still nullable, and Postgres did exactly what it was told: stored NULL.

the find

The refactor was correct for the problem it set out to solve. Computing the cycle length per plan inside the ORM is the right design once a single fixed default stops covering every plan. The gap was scope: "move the default into the application" was reviewed and tested against every ORM write path, and there are plenty of those, but nobody enumerated the write paths that skip the ORM entirely. The reactivation importer had worked unmodified for over a year specifically because it never had to know what the default was. It depended on the database doing that job, and the migration quietly took that job away.

The NULL didn't surface at insert time because the column stayed nullable. It didn't surface at invoice time either, because of a defensive coalesce added two years earlier for an unrelated edge case:

amount_due = coalesce(daily_rate * sub.billing_cycle_days, 0)

That line was written to stop a crash when daily_rate was occasionally missing for trial accounts. It had the side effect of turning any NULL in billing_cycle_days into a quiet, valid-looking $0.00 invoice instead of an error. Two independent decisions, a dropped default and an old defensive coalesce, combined to turn a missing column value into 214 invoices nobody caught for eleven days.

the fix

Immediate: backfill the 214 affected subscriptions using the same plan-to-cycle mapping the ORM default now uses, then regenerate and reissue the affected invoices as corrected line items rather than duplicate charges:

UPDATE subscriptions SET billing_cycle_days = CASE plan_id
    WHEN 'flex' THEN 7
    WHEN 'custom' THEN NULL  -- requires manual contract lookup, flagged separately
    ELSE 30
END
WHERE billing_cycle_days IS NULL;

Structural fix: re-add a database-level default as a safety net alongside the application-level one, so any write path that doesn't know about plan-specific cycles still gets a sane fallback instead of silence, and add a NOT NULL constraint so a future gap fails loudly at insert time instead of storing a placeholder value:

def upgrade():
    op.execute("UPDATE subscriptions SET billing_cycle_days = 30 WHERE billing_cycle_days IS NULL")
    op.alter_column(
        'subscriptions', 'billing_cycle_days',
        server_default='30',
        nullable=False,
    )

The defensive coalesce in the invoicing job came out too. A NULL cycle length is now a fatal error that pages on-call before an invoice gets created, not a silent $0.00 row that only shows up in a revenue dashboard eleven days later. Process fix: the migration review checklist now has one explicit line for any PR touching a server_default, asking whether a raw SQL, bulk COPY, or ETL write path depends on it. The reactivation script also got a column list update to pass billing_cycle_days explicitly going forward, instead of depending on either layer's default.

the aftermath

The refactor itself was good work. The mistake was treating "every write path" as a given instead of a list, and a script that had gone untouched for a year was exactly the kind of write path that didn't make it onto anyone's list.