How 1,000 Mapped Fields in Elasticsearch Made New Listings Invisible for 11 Days

03:12 UTC. One support ticket: "my listing isn't showing up in search." Not urgent-looking, not a spike, just one seller who'd listed a guitar amp an hour earlier and couldn't find it. By the time the support queue opened at 09:00, there were six more, all different sellers, all different categories, all the same complaint. Nothing in Grafana was red.

the scramble

First theory: relevance, not absence. New listings sometimes rank low until they accumulate views and clicks, so "can't find it" often just means "page three." Someone pulls up one of the reported listing IDs and searches its exact title, word for word. Zero results. Not buried, gone.

Second theory: the Kafka consumer that feeds the search indexer has fallen behind. The listings service writes to Postgres and emits a listing.created event; a consumer picks it up and upserts the document into Elasticsearch. If that consumer were lagging, new listings would just take longer to appear. Consumer group lag:

TOPIC            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
listing-events   0          8842210         8842213         3
listing-events   1          8841998         8842001         3
listing-events   2          8842055         8842055         0

Lag in single digits. The consumer is keeping up in real time. Whatever is eating these listings, it isn't a backlog, and it isn't happening somewhere between Kafka and the indexer process. It's happening inside the upsert itself, and the dashboards that would normally show an indexing problem, consumer lag, CPU, cluster health, were all green because none of them measure "did the write actually succeed."

the hunt

Cluster health was green because nothing about the cluster was unhealthy. The problem had to be in the indexer's own application logs, not the infra dashboards, so the next step was grepping the indexer's log stream directly for the IDs from the support tickets:

WARN  index_upsert_failed listing_id=lst_8f2k1 index=listings
  illegal_argument_exception: Limit of total fields [1000] in index [listings]
  has been exceeded

Every missing listing had this exact line, same error, same index, logged at WARN. Nobody had an alert on indexer warn-level logs, because the indexer's retry handler had been written specifically to survive one bad document without blocking every listing queued behind it:

async function upsertListing(doc) {
  try {
    await es.index({ index: 'listings', id: doc.id, document: doc });
  } catch (err) {
    log.warn('index_upsert_failed', { listing_id: doc.id, error: err.message });
    // deliberately swallow: one bad document shouldn't stall the
    // consumer for every listing behind it in the partition
  }
}

That design made sense for its original purpose, which was surviving the occasional malformed payload from a buggy client. It had no way to distinguish "this one document is broken" from "every future document with a new attribute will fail the exact same way, forever, until the mapping is fixed." The first is transient. The second isn't, and it had been happening to a growing share of new listings since a feature shipped three weeks earlier: Custom Specs, which let sellers attach arbitrary key-value attributes to a listing (battery_life_hours, oak_finish, frame_size_cm), free text on both sides, indexed dynamically so sellers' custom fields would be searchable without anyone pre-defining a schema for them.

Elasticsearch's dynamic mapping creates a new mapped field the first time it sees a key it hasn't seen before, and it does this cumulatively across every document ever indexed into that index, not per document. A listing with three custom spec keys doesn't cost three fields of headroom and give them back when the listing is deleted; it permanently claims three slots against the index's total field budget, shared by every seller who will ever list anything. The default budget, index.mapping.total_fields.limit, is 1,000:

# field count via _field_caps, sampled daily for the three weeks after launch
Oct 13: 412 fields
Oct 20: 781 fields
Oct 27: 1,000 fields  <- cap reached, first silent rejection
Nov 2:  1,000 fields  <- still capped, 11 days of rejections by now

Once every sellers' worth of unique attribute names pushed the index past 1,000 distinct fields, any listing introducing even one more new key failed outright. Listings reusing only already-seen keys kept indexing fine, which is exactly why this looked random from the support queue: some sellers hit it immediately, others didn't hit it for days, depending entirely on whether their particular combination of spec names happened to be new to the index.

the find

Custom Specs mapped seller-defined attributes as regular dynamic object fields. Elasticsearch mapped a new field for every unique key ever seen, cumulatively, and the index crossed its 1,000-field cap eleven days before anyone noticed. Every indexing call that would have added a 1,001st field failed with illegal_argument_exception, and the indexer's own error handling, built to protect throughput against one-off bad documents, logged that failure at a level nobody was watching and moved on to the next message.

the fix

The real fix was changing how seller-defined attributes get mapped at all. Elasticsearch has a type built for exactly this shape of data, arbitrary key-value pairs that shouldn't each become their own mapped field: flattened. A flattened field is indexed and searchable as a single field no matter how many distinct keys live inside it.

PUT /listings-v2/_mapping
{
  "properties": {
    "custom_specs": { "type": "flattened" }
  }
}

That required a reindex into a new index and an alias swap rather than an in-place mapping edit, since Elasticsearch doesn't let you change an existing field's type. Second, the swallowed exception became a counted, alertable failure instead of a silent one:

async function upsertListing(doc) {
  try {
    await es.index({ index: 'listings', id: doc.id, document: doc });
  } catch (err) {
    log.error('index_upsert_failed', { listing_id: doc.id, error: err.message });
    indexUpsertFailures.inc({ error_type: err.name });
    // still don't block the consumer on one bad document, but now
    // a sustained failure rate pages someone instead of hiding in WARN
  }
}

A Prometheus alert on indexUpsertFailures sustaining above a low threshold for more than five minutes now pages the on-call team directly, which is the gap that let this run for eleven days in the first place: the failure rate was never zero, so a threshold alert would have caught it on day one instead of waiting for a support ticket.

the aftermath

A backfill script re-ran every affected listing ID against the new flattened mapping, and all 1,340 were searchable again within the hour. No listing had been lost; Postgres had every one of them the whole time. They were simply invisible to the one surface buyers actually use to find anything.

Eleven days of a growing rejection rate, and the first signal anyone outside the indexer's own log stream saw was a seller asking why their guitar amp wasn't showing up.