Context Window Truncation Safeguards: Stop AI Agents From Forgetting Critical Instructions

A context window is a capacity limit, not a memory system. When a long-running agent crosses that limit, something must be removed, compressed, or retrieved later. If your runtime makes that choice invisibly, the agent can keep producing fluent answers after it has forgotten the rule that made the workflow safe.

The fix is to treat context as a budgeted data structure. Separate durable invariants from conversational history, measure what is admitted, and test the exact behavior at the truncation boundary.

1. Classify every context item

Start with four classes:

• Invariants: authorization limits, user constraints, safety rules, output contracts.
• Task state: accepted inputs, current step, completed effects, pending approvals.
• Evidence: tool results and source excerpts needed for the next decision.
• Conversation: prose that helps tone or continuity but can be summarized.

Do not put these classes into one undifferentiated message list. An invariant should never compete with ten pages of logs for the same eviction policy.

type ContextItem = {
  id: string;
  kind: "invariant" | "state" | "evidence" | "conversation";
  priority: number;
  tokenEstimate: number;
  expiresAfterStep?: number;
  contentHash: string;
};

Expected result: the runtime can explain why each item is present and what may be dropped.

2. Reserve tokens before model invocation

Do not fill the window to the advertised maximum. Reserve capacity for the model’s answer, tool schemas, and serialization overhead.

const budget = {
  modelLimit: 128_000,
  outputReserve: 8_000,
  toolReserve: 12_000,
  safetyMargin: 4_000,
};

const inputLimit = budget.modelLimit
  - budget.outputReserve
  - budget.toolReserve
  - budget.safetyMargin;

Tokenizers differ across models. Use the provider’s tokenizer when available, then keep a margin for hidden wrappers and schema expansion. A character-count estimate is acceptable for admission control only if it is calibrated and conservative.

3. Pin invariants by reconstruction

“Pinned” should mean the runtime reconstructs the invariant block on every call. It should not mean the instruction happened to appear near the start of an old transcript.

Build the request in a deterministic order:

1. system policy and authorization envelope;
2. normalized task contract;
3. durable workflow state;
4. current evidence;
5. compact conversation summary;
6. latest user turn.

Store a hash of the invariant block with each run. If the hash changes unexpectedly, fail closed or route for review.

4. Summarize state, not obligations

Summaries are lossy. Never ask a model to compress a safety rule and then rely on its paraphrase as the rule itself. Keep obligations verbatim or as structured data.

Good summary input includes prior discussion, discarded options, and old evidence. Durable state should be written as facts:

{
  "goal": "publish three approved records",
  "completedIds": ["a17", "a18"],
  "pendingIds": ["a19"],
  "externalWritesAllowed": false,
  "approvalRequiredFor": ["send", "delete"]
}

Generate summaries at stable checkpoints rather than only after overflow. Store the source-message range and the summarizer model so a bad summary can be audited.

5. Retrieve evidence narrowly

A vector store is not a license to dump every similar chunk into the prompt. Retrieve by task ID, source authority, recency, and semantic relevance. Cap both chunk count and total tokens.

Prefer exact records for decisions: API responses, database rows, and signed approvals. Use semantic retrieval for supporting prose. If two retrieved facts conflict, preserve both and surface the conflict instead of letting rank order decide truth.

6. Make truncation observable

Emit a context manifest before each model call:

type ContextManifest = {
  runId: string;
  inputTokens: number;
  limit: number;
  includedIds: string[];
  droppedIds: string[];
  summaryIds: string[];
  invariantHash: string;
};

Alert when an invariant or required state item is missing, when summary compression exceeds a threshold, or when evidence was dropped from a decision step. Do not log sensitive prompt content by default; hashes and stable IDs often provide enough operational visibility.

7. Test the boundary deliberately

Most teams test with short chats and discover truncation behavior in production. Build fixtures that place a critical instruction at the beginning, middle, and end of a context near the model limit. Add irrelevant material until eviction occurs.

Assert that:

• the reconstructed invariant remains present;
• completed side effects remain recorded;
• the agent refuses an action outside its authority;
• conflicting evidence is surfaced;
• the summary retains exact IDs and amounts;
• retries do not repeat committed effects.

The “lost in the middle” research shows why placement matters even before hard truncation. A message can technically remain in context yet receive weak attention. Repeat the normalized task contract close to the decision point without creating contradictory copies.

8. Fail closed on unsafe compression

If the runtime cannot fit required invariants, state, tool schema, and current evidence, the correct response is not aggressive summarization. Split the task, ask for a narrower scope, or use a model with a larger supported context.

For autonomous workflows, mark the run blocked_context_budget and preserve the checkpoint. A visible pause is cheaper than a fluent action taken after the authorization clause disappeared.

Production checklist

• Define a token budget per model and tool set.
• Reconstruct invariants every call.
• Store task state outside the transcript.
• Summarize conversation separately from obligations.
• Log manifests without leaking prompt content.
• Test attention and hard-truncation boundaries.
• Block execution when required context cannot fit.

Context windows will keep growing, but so will agent workloads. Reliability comes from deciding what the model must know at each step—not from assuming a larger window remembers everything equally well.