Context Window Budgeting for LLM Apps: Stop Truncating the Evidence

Your retrieval system finds the correct policy. Your conversation history contains the customer’s exception. Your tool schema explains the only allowed action. Then the runtime trims the beginning of the prompt to fit the model—and quietly removes all three.

A bigger context window delays this bug. It does not fix it.

Context budgeting treats tokens as capacity that must be allocated before the prompt is assembled. The goal is not to stuff the maximum amount of text into every call. It is to preserve the evidence and instructions that the decision actually depends on.

Reserve Capacity Before Collecting Context

Start with the provider’s advertised window, then subtract an output reserve. The reserve should include the expected answer, tool-call arguments, reasoning overhead when applicable, and a safety margin for tokenizer mismatch.

Divide the remaining input budget into explicit buckets:

• system and safety policy
• tool definitions and response schemas
• current user request
• conversation state
• retrieved evidence
• tool observations
• formatting examples

Set hard minimums for policy, request, and required schemas. Flexible sections can compete for what remains. If the total still does not fit, the runtime should enter a visible degradation mode rather than calling the model with accidental truncation.

Count tokens with the tokenizer for the selected model whenever possible. Character estimates are useful for early filtering but unreliable around code, non-Latin text, serialized JSON, and long identifiers.

Rank Evidence by Decision Value

Retrieval score is not the same as importance. A low-scoring paragraph may contain the exception that changes the answer. Add features for source authority, freshness, user permissions, contradiction, requested date range, and whether a passage supports a required citation.

Group evidence by claim. Keeping five near-duplicate passages about the default rule while dropping the only exception wastes the window. Use diversity constraints and retain provenance so summaries can point back to originals.

Position matters. Long-context research has shown that models may use information differently depending on where it appears. Place the current task and decisive evidence in consistent, high-attention locations. Do not assume any text is safe merely because it technically fits.

If documents exceed their bucket, extract sections relevant to the query before summarization. A summary that erases conditions, dates, units, or negation is not compression; it is evidence corruption.

Compress State Without Inventing It

Conversation memory should be structured. Keep durable facts, unresolved questions, approvals, tool results, and user preferences as separate fields with sources. Summarize social chatter more aggressively than commitments or numbers.

Use rolling summaries with checkpoints. Before replacing older messages, compare the proposed summary against a list of protected facts. Preserve exact identifiers, quoted user constraints, money, dates, and the distinction between requested, attempted, and completed actions.

Tool output should be normalized before entering context. Strip transport metadata and repeated fields, but keep status codes, partial failures, pagination, and uncertainty. Never compress three of five records updated into update complete.

Cache stable prefixes when the provider supports it. Caching can lower cost and latency, but cached tokens still occupy context. It is an economic optimization, not extra capacity.

Fail Explicitly When the Answer Does Not Fit

Measure budget decisions in production: tokens requested per bucket, evidence dropped, summary generations, truncation events, output reserve used, and answer quality after compression.

Create adversarial tests where the decisive fact appears in an old turn, a low-ranked result, the middle of a long document, or a conflicting source. Assert that the assembler either preserves it or returns a typed INSUFFICIENT_CONTEXT_BUDGET result.

When capacity is genuinely insufficient, split the work. Retrieve in stages, ask a clarifying question, create a verified intermediate artifact, or use a map-reduce pattern with claim-level checks. Do not hide the overflow by slicing strings.

The context window is working memory, not a data lake. Reliable LLM systems decide what deserves that memory, record what was excluded, and refuse to pretend that truncated evidence was never relevant.