How Caching DataLoader Instances Leaked Salary Bands Across 58 Orgs for 11 Days

09:14 UTC. A Zendesk ticket lands in the on-call queue, tagged security. Subject: "I can see comp bands for my whole team and I'm not a manager." Screenshot attached: the team directory page, logged in as an individual contributor, showing a column that shouldn't exist for that role at all.

the setup

people-api is the GraphQL service behind the team directory feature, an Apollo Server instance sitting in front of Postgres. Org admins can see a compBand field on each team member; everyone else gets name, title, and email. That check used to live inline in the resolver, straightforward and correct.

Eleven days earlier, a performance pass flagged people-api as one of the biggest sources of GC pauses under load. A --prof run pointed at DataLoader construction: thousands of fresh instances a second, one per request, each holding its own closure and internal cache that got garbage collected a few hundred milliseconds later. An engineer traced it to createContext, saw a brand-new DataLoader built on every request, and reasoned that requests hitting the same org didn't need to redo that work. The fix shipped as a one-line-feeling change: memoize the loader per org ID in a module-level map.

the scramble

First theory: a frontend permission bug, the UI rendering a column it should be hiding for this role. On-call opened the network tab on the reporter's session. The raw GraphQL response body already contained compBand for every team member. Whatever this was, it wasn't a rendering problem, the API itself was handing the field out.

Second theory: a bad JWT, the token carrying the wrong role claim for this user. On-call decoded it directly.

{
  "sub": "usr_88213",
  "org_id": "org_4471",
  "role": "member",
  "exp": 1788123456
}

Role was correctly "member," not "admin." The token was fine. Whatever was leaking the field, it wasn't happening at the auth layer.

Third theory: someone had deleted the redaction check in a recent resolver change. On-call pulled up the resolver file directly, expecting to find the guard clause missing.

orgMembers: async (_parent, { ids }, context) => {
  const loader = getOrgMemberLoader(context.orgId, context.user.role);
  return loader.loadMany(ids);
},

The check was there, just one level down, inside getOrgMemberLoader. Nothing obviously wrong at the call site. Three theories, three dead ends, and the field was still showing up in fresh sessions on every retry.

the hunt

With the resolver itself clean, the next stop was the loader function it was calling into. That's where the GC-pressure fix from eleven days ago lived.

const orgMemberLoaders = new Map();

function getOrgMemberLoader(orgId, role) {
  if (!orgMemberLoaders.has(orgId)) {
    orgMemberLoaders.set(orgId, new DataLoader(async (ids) => {
      const rows = await db.orgMembers.findByIds(orgId, ids);
      return rows.map((row) => (role === 'admin' ? row : redactCompBand(row)));
    }));
  }
  return orgMemberLoaders.get(orgId);
}

The map is keyed on orgId alone. The role argument only matters the first time a given org's loader gets created, because that's the only time the batch function itself gets constructed, closure and all, everything after that reuses the same instance and the same baked-in redaction decision, no matter who's calling.

To confirm, on-call added a counter around the batch function and pulled the last four hours of invocations for org_4471 from Datadog.

people_api.org_member_loader.batch_fn_invoked{org_id:org_4471}
  05:02 UTC  1 invocation
  05:02–09:14 UTC  0 invocations, 214 loadMany() calls served from cache

One real database fetch at 05:02, then 214 requests over the next four hours served entirely from the cached loader without ever running the batch function again. Whoever made that 05:02 request decided, for everyone at that org, whether comp bands would be visible for the rest of that server process's life.

05:02:11 org_4471 usr_10029 role=admin query=orgMembers status=200

An admin loaded the team directory at 05:02 to check a headcount number. That request warmed the loader with unredacted rows. Every teammate who opened the same page afterward, on whichever pod happened to still be holding that cached instance, got the admin's view back.

the find

Root cause: a GC-pressure fix moved DataLoader construction from per-request, inside Apollo's context factory, to a module-level map keyed only by orgId. Field-level redaction was implemented inside the batch function's closure instead of the resolver, so it ran exactly once per org, whenever the first request happened to arrive, and that first caller's role determined what every subsequent caller received for as long as that server process kept the loader cached. Non-admin requests weren't failing an authorization check, they were never running one, they were reading a response shaped by someone else's permissions.

the fix

Stop-the-bleeding move first: a rolling restart of every people-api pod, which cleared every cached loader instance and forced the next request per org to rebuild from a clean state. That bought time, not a fix, the same bug would recur the moment two requests with different roles hit the same org within one process lifetime.

The actual fix separates the two things that had gotten fused together: batching, which is safe to cache because raw rows don't vary by caller, and redaction, which depends entirely on who's asking and has to run fresh every time. The loader goes back to fetching only, no role logic inside it at all, and the resolver applies the field mask itself, per request, regardless of whether the underlying rows came from a warm cache or a fresh query.

export function createContext({ req }) {
  return {
    user: req.user,
    orgId: req.user.orgId,
    loaders: {
      orgMembers: new DataLoader(async (ids) => {
        return db.orgMembers.findByIds(req.user.orgId, ids);
      }),
    },
  };
}

orgMembers: async (_parent, { ids }, context) => {
  const rows = await context.loaders.orgMembers.loadMany(ids);
  return rows.map((row) =>
    context.user.role === 'admin' ? row : redactCompBand(row)
  );
},

The loader is created fresh in the context factory again, so it never outlives a single request, and the redaction that had briefly lived inside the batch function moved out entirely. Even if someone reintroduces per-org caching later for a real performance reason, this shape survives it, because authorization no longer depends on which request happened to build the cache.

The GC pressure that started all this got solved separately and correctly: batching multiple orgMembers calls within a single request using DataLoader's normal per-tick batching window, instead of trying to share instances across requests. That cut allocation counts without ever touching who gets to see what.

the aftermath

Affected orgs got a direct notice naming the exact field exposed and the exact window. No other people-api field was affected, the redaction bug was specific to compBand, the only field whose visibility ever depended on the caller instead of the data.

billing-api and auth-svc use their own DataLoader instances, built the same per-request way this one used to be. Neither was touched by this change and neither saw a single leaked field.