Agents in production, part 3: the 2,000-person enterprise
A 2,000-person company had thirty teams running agents in production. Each team had independently implemented the patterns from Parts 1 and 2 — async queues, circuit breakers, step-level tracing. In Q3, the company's LLM spend tripled in sixty days. The CFO sent a spreadsheet. The platform team had no answers. Every team said their usage was flat. Nobody was lying. Nobody could see the other twenty-nine teams.
The investigation revealed a cascade: one team's document processing agent had shipped a new feature that doubled its token consumption. That team's increased usage pushed the org-wide account into a higher rate limit tier. Three other teams, already running near their effective throughput ceiling, started seeing elevated 429 rates. Their circuit breakers opened. Jobs backed up in their dead letter queues. Engineers escalated. The platform team increased the rate limit allocation across the board. Spend doubled again.
This is part 3 of the series. The patterns from Parts 1 and 2 work inside a single team. They break at enterprise scale because the problems are not technical — they are coordination problems in a technical disguise.
What each team already has right — and why it is not enough
At a 2,000-person company with thirty agent-running teams, every team has done the work. Async pipelines. Checkpointed step runners. Per-step trace spans. Output validation. The individual team architectures are solid.
The gap is at the boundaries between teams. Three failure modes dominate:
Shared quota with no allocation. The company has one set of LLM API credentials shared across thirty teams. The rate limit is an org-level ceiling, not a per-team ceiling. When team A increases throughput by 40%, teams B through D each lose part of their effective capacity — but none of them know which team caused the degradation. Their circuit breakers open. They file tickets with the platform team. The platform team increases the org limit and the underlying problem remains invisible.
No agent catalog. There is no authoritative list of which agents exist, who owns them, what data they access, or what their production throughput is. When a data compliance audit asks "which systems access customer PII through an LLM?", the answer requires manually surveying thirty engineering managers. The audit takes three weeks. Three agents are discovered during the survey that nobody had formally registered.
No cost attribution. The monthly LLM invoice is a single line item. The platform team can see total tokens. They cannot see which team, which agent, which environment, or which feature flag drove a specific cost spike. The CFO spreadsheet has one number. Engineers have anecdotes.
// What 30 teams each doing this produces at the org level
const response = await openai.chat.completions.create({
model: 'gpt-4o',
messages: [...],
// No team tag. No agent tag. No env tag.
// 30 teams × N agents × M calls = unattributable spend
});
// vs. what every call should look like
const response = await openai.chat.completions.create({
model: 'gpt-4o',
messages: [...],
}, {
headers: {
'X-Agent-Team': 'docs-processing',
'X-Agent-Name': 'contract-extractor',
'X-Agent-Env': 'production',
'X-Trace-Id': ctx.traceId,
}
});
// Paired with a proxy that captures this and emits per-call metrics
The corrected architecture: platform-layer controls
The enterprise patterns are not more complex versions of the team-level patterns. They are a different layer entirely: infrastructure that teams use rather than infrastructure teams build themselves.
1. An LLM gateway with per-team rate limit allocation. Instead of giving all thirty teams the same API key, the platform team runs a thin proxy that enforces per-team rate limits, adds attribution headers, and records every call. Teams call the gateway, not the LLM vendor directly. The gateway holds the vendor credential.
// llm-gateway: central proxy with per-team enforcement
import Fastify from 'fastify';
import { RateLimiter } from 'limiter';
const teamLimits: Record<string, RateLimiter> = {
'docs-processing': new RateLimiter({ tokensPerInterval: 100_000, interval: 'minute' }),
'customer-support': new RateLimiter({ tokensPerInterval: 60_000, interval: 'minute' }),
'data-enrichment': new RateLimiter({ tokensPerInterval: 80_000, interval: 'minute' }),
// ...one entry per registered team
};
fastify.post('/v1/chat/completions', async (req, reply) => {
const team = req.headers['x-agent-team'] as string;
const limiter = teamLimits[team];
if (!limiter) {
return reply.status(403).send({ error: 'Unregistered team — register in the agent catalog first' });
}
const allowed = await limiter.removeTokens(estimateTokens(req.body));
if (!allowed) {
metrics.increment('gateway.rate_limited', { team });
return reply.status(429).send({ error: 'Team rate limit exceeded', retryAfter: 60 });
}
// Forward to vendor, emit per-call metrics
const response = await forwardToOpenAI(req.body, req.headers);
metrics.histogram('gateway.tokens_used', response.usage.total_tokens, { team });
return reply.send(response);
});
The gateway does three things at once: enforces allocation (team A's spike does not affect team B), attributes spend (every token has a team owner), and surfaces the catalog enforcement point (unregistered agents cannot call the LLM).
2. An agent catalog as the source of truth. Before an agent can call the LLM gateway, it must be registered. The catalog record contains the team, the agent name, the environments it runs in, the data sources it accesses, the approved models, and the monthly token budget. Registration is lightweight — a YAML file in a shared repo with a one-hour review SLA from the platform team.
# agent-catalog/teams/docs-processing/contract-extractor.yaml
name: contract-extractor
team: docs-processing
owner: eng-lead@company.com
environments: [staging, production]
data_access:
- source: contracts-database
classification: confidential
approved_by: data-privacy-team
approved_at: 2026-04-15
models:
approved: [gpt-4o, gpt-4o-mini]
default: gpt-4o-mini
budget:
monthly_tokens: 50_000_000
alert_threshold: 0.8 # alert at 80% of budget
sla:
p99_latency_ms: 15000
error_rate_threshold: 0.02
The catalog is not bureaucracy. It is the answer to every compliance audit question. When legal asks "which systems access customer contracts through an LLM?", the answer is a grep of the catalog. The three-week audit becomes a five-minute query.
3. Cost attribution dashboards. With the gateway emitting per-call metrics tagged by team and agent, the platform team builds a single dashboard: daily spend by team, by agent, by model, by environment. Budget alerts fire at 80% of a team's monthly allocation. The CFO spreadsheet has thirty rows instead of one.
// Cost attribution — what the platform team can now query
SELECT
team,
agent_name,
model,
DATE_TRUNC('day', called_at) AS day,
SUM(total_tokens) AS tokens,
SUM(total_tokens) * 0.000015 AS cost_usd -- gpt-4o rate
FROM llm_gateway_calls
WHERE env = 'production'
AND called_at >= NOW() - INTERVAL '30 days'
GROUP BY 1, 2, 3, 4
ORDER BY cost_usd DESC;
-- Output when the cost spike happened:
-- docs-processing | contract-extractor | gpt-4o | 2026-06-14 | 8.2M tokens | $123
-- docs-processing | contract-extractor | gpt-4o | 2026-06-15 | 16.1M tokens | $242 ← spike
-- Previous day was normal. Feature flag shipped June 14.
4. A governance review for data access changes. The single highest-risk action at enterprise scale is an agent gaining access to a new data classification. A support agent that previously read public FAQ articles being updated to read customer billing records is not a code change — it is a compliance event. The catalog enforces that every new data source requires a pull request review from the data privacy team before the gateway allows calls from that agent environment.
// Enterprise agent architecture
Team A agents ──┐
Team B agents ──┤ ┌─────────────────────────┐
Team C agents ──┼───────────────►│ LLM Gateway │──► OpenAI / Anthropic
... │ per-team │ rate limits per team │ (API key stored here only)
Team Z agents ──┘ rate limit │ attribution headers │
enforcement │ usage metrics per call │
└────────────┬────────────┘
│ rejects unregistered teams
┌────────────▼────────────┐
│ Agent Catalog │
│ YAML per agent │
│ team + data approvals │
│ model allowlist + budget│
└────────────┬────────────┘
│
┌────────────▼────────────┐
│ Cost Attribution Board │
│ spend by team / agent │
│ by model / env / day │
│ budget alerts at 80% │
└─────────────────────────┘
What changed when the platform team shipped this
The spend tripling was solved in two weeks, not by reducing usage but by making it visible.
Once the gateway was in place and per-team dashboards were live, the docs-processing team
immediately saw their contract-extractor agent was calling gpt-4o on every
request. They switched non-critical paths to gpt-4o-mini. Spend dropped 35%
with no change to output quality for 80% of their workload.
The rate limit cascade stopped because team allocations were now enforced independently. Team A's spike no longer affected teams B through D. Each team had a stable ceiling. Circuit breaker trip rates dropped to near zero across the three affected teams within a week.
The compliance audit that had previously taken three weeks closed in forty minutes. The catalog had thirty-one registered agents. Three were in staging only. Twenty-eight were in production. Seven accessed data classified as confidential or above, all with approval records attached.
The pattern across all three parts
Looking at the series as a whole: the 20-person startup's problem was treating the LLM as a fast local function. The 150-person scale-up's problem was treating the multi-step chain as an opaque unit. The 2,000-person enterprise's problem was treating thirty independent agent deployments as thirty independent problems.
Each failure is a boundary problem. The startup had no boundary between the request handler and the LLM. The scale-up had no boundary between pipeline steps. The enterprise had no boundary between teams' agents and the shared infrastructure beneath them.
The patterns in each part add a boundary — and with it, a place to enforce reliability, observe behavior, and control cost. The architecture is not about the LLM. It is about knowing where your system ends and someone else's begins.