Token Budget SLOs for LLM Apps: Control Cost Without Breaking Quality

LLM cost problems rarely begin with a dramatic price increase. They begin with five extra retrieved documents, an agent loop that calls the model twice more, and a verbose tool result copied into every turn.

The request still costs pennies. At scale, the margin disappears.

A token budget service-level objective makes cost a designed property of the workflow. It defines how much input, output, and repeated context a successful task may consume, then links overruns to quality-aware controls.

Budget the Task, Not the Request

A single model request is often the wrong unit. A support-resolution task may include classification, retrieval, three tool calls, a repair pass, and a final response. Budget the complete customer outcome.

For each workflow, record:

• input and output tokens by step;
• cached versus uncached input where the provider exposes it;
• tool-result bytes before and after normalization;
• retry and repair tokens;
• model and region pricing version;
• whether the task actually succeeded.

Cost per successful task is more useful than cost per call. A cheap model that needs repeated repairs may be more expensive than a stronger model that finishes once.

Define separate distributions for ordinary and complex cases. One global average hides the long tail that produces both high bills and bad latency.

Write a Real SLO

An SLO needs a window, population, threshold, and quality condition. For example: “Over 28 days, 99% of eligible invoice-extraction tasks complete within 18,000 total tokens while maintaining at least 98.5% field-level accuracy.”

That is better than “keep token use low.” It tells the team which tasks count and prevents cost reduction from being declared successful after accuracy collapses.

Use multiple indicators when needed:

• median and p95 tokens per successful task;
• output-to-input ratio;
• retrieval tokens per cited fact;
• tokens consumed by failed attempts;
• percentage of tasks crossing a hard ceiling;
• dollar cost per resolved customer outcome.

Create an error budget for overruns. A temporary increase may be acceptable during an incident or a difficult migration. A persistent burn rate signals structural context growth.

Reserve Output Before Building Context

Context windows create a dangerous illusion: if text fits, it is affordable and useful. Reserve the required output capacity first, then allocate the remaining input budget among instructions, conversation, retrieval, and tool results.

const budget = {
  total: 24_000,
  outputReserve: 2_500,
  system: 2_000,
  history: 5_000,
  retrieval: 10_000,
  tools: 4_500,
};

Enforce each compartment before assembling the prompt. If retrieval overflows, rerank, deduplicate, or summarize at the source boundary. Do not silently steal the output reserve and then blame the model for truncation.

Count with the target model’s tokenizer when possible. Character estimates are useful for early rejection but inaccurate across languages, code, and structured data.

Stop Paying for Repeated Context

Agent architectures often resend static instructions, schemas, and growing histories on every turn. Identify what is immutable, what can be referenced by ID, and what should be compacted.

Normalize tool results into the fields the next decision needs. A 50-kilobyte API response should not enter context because one nested status matters. Keep the raw result in durable storage and put a typed, provenance-linked projection into the prompt.

Conversation compaction must preserve decisions, unresolved obligations, identifiers, and safety constraints. A beautiful prose summary that drops “do not contact this customer” is not compression; it is corruption.

Prompt caching can lower price and latency for repeated prefixes, but it does not remove architectural waste. Track cached tokens separately so a provider pricing change does not reveal an unaffordable design.

Degrade Gracefully

When a task approaches its budget, choose a deliberate fallback. Options include using fewer retrieved documents, switching an exploratory step to a smaller model, asking the user for a missing discriminator, skipping optional analysis, or escalating.

Never remove safety instructions, authorization checks, or evidence required for a high-impact action. Budget pressure should shrink optional breadth before it weakens control.

Give agent loops a remaining-budget value and a maximum number of transitions. The agent should know that another search or repair has a cost. It should not be allowed to negotiate its own hard ceiling.

if (spentTokens + estimatedNextStep > hardLimit) {
  return escalate({ reason: "token_budget_exhausted", traceId });
}

Distinguish a budget stop from a model failure in telemetry. Otherwise teams retry the same task and double the cost.

Route by Economics and Risk

Model routing should consider task value, uncertainty, and consequence—not price alone. A low-value summarization can use aggressive caps. A complex contract review may justify more context but require explicit customer pricing or internal approval.

Use evaluations to build a quality-versus-cost frontier. Test models and prompt variants on the same cases, including long-tail examples. Select the cheapest configuration that meets the required quality and safety threshold for that workflow.

Do not route based solely on user plan if the action risk is identical. A cheaper tier still needs safe behavior.

Make Overruns Debuggable

Emit a trace event for every budget allocation, truncation, compaction, retry, and fallback. Dashboard by workflow version, model, customer segment, language, and tool.

The most actionable alert says “p95 retrieval tokens for catalog support increased 42% after index v17,” not “AI cost is high.” Link spend changes to deployments and data-volume changes.

Keep prices in a versioned table rather than hard-coding dollars into historical events. Store raw token counts so finance can recompute cost when contracts or provider prices change.

Protect Quality While Cutting Cost

Cost optimization is an evaluation discipline. Remove duplicate context, narrow schemas, and reduce unnecessary loops before making the model weaker. Measure task success after every change.

Token budget SLOs create a useful constraint: the product must deliver a defined outcome within a known envelope. That turns cost from a month-end surprise into an engineering signal—and makes sustainable AI margins possible.