LLM Context Window Management: Keep Production Agents Fast, Cheap, and Coherent

If your agent gets slower every turn, you probably do not have a model problem. You have a junk-drawer problem.

This guide turns llm context window management production from a vague prompt-engineering headache into a repeatable system: budget the tokens, rank what matters, summarize what ages, retrieve what is needed, and cache the boring stable stuff.

The context window is not free storage. It is expensive working memory. Treat it like a database table with a brutal size limit, not like a magical attic where every Slack thread, PDF chunk, tool result, and user ramble gets to live forever.

Prerequisites

You do not need to be a hardcore ML engineer. You do need a basic agent stack.

Before you start, have:

An LLM API provider with token usage metadata in responses
A server-side place to store conversation state, such as Postgres, Redis, DynamoDB, or your framework’s checkpointer
A tokenizer or token-estimation utility for your target model
A retrieval store if your agent needs documents, tickets, policies, code, or user records
Logs for latency, input tokens, output tokens, errors, and model calls

Example dependencies for a TypeScript service:

npm install js-tiktoken zod

If you use LangChain, LlamaIndex, Mastra, Vercel AI SDK, or your own wrapper, the pattern is the same. The exact method names change. The architecture does not.

The Production Context Rule

Your agent should never blindly send the entire conversation.

That is the amateur move. It works in demos because demos are short. In production, conversations get long, tool outputs get chunky, users paste garbage, and your beautiful agent starts paying premium token rates to reread yesterday’s irrelevant mess.

Use this rule instead:

Send the smallest context packet that can answer the current request correctly.

That packet usually contains five layers:

Stable instructions
User and account state
Task-specific retrieved context
Recent conversation turns
A compact summary of older conversation

The mistake is treating all context as equal. It is not.

A system policy is more important than a greeting from 40 turns ago. A failed tool call from two minutes ago may matter more than a 3,000-token help document. A user’s billing plan might matter for every answer. Their typo from last week probably does not.

Step 1: Create A Token Budget

Start with a budget before writing clever memory logic. Otherwise you are just vibes-testing your infrastructure.

Example budget for an agent using a model with a large context window:

type ContextBudget = {
  maxInputTokens: number;
  reservedOutputTokens: number;
  systemTokens: number;
  memoryTokens: number;
  retrievalTokens: number;
  recentMessagesTokens: number;
};

export const supportAgentBudget: ContextBudget = {
  maxInputTokens: 64000,
  reservedOutputTokens: 4000,
  systemTokens: 4000,
  memoryTokens: 6000,
  retrievalTokens: 30000,
  recentMessagesTokens: 20000,
};

Do not allocate 100% of the window. Leave room for output, tool-call arguments, provider-specific overhead, and weird user input. A good starting point is to use 70-85% of the advertised context window as your real ceiling.

Expected result: every request has a hard limit before it touches the model. No surprise 400 errors. No runaway cost because one user pasted a 90-page contract into chat.

Step 2: Rank Context By Value

Use priority buckets. This is boring. Boring is good. Boring survives production traffic.

type ContextItem = {
  id: string;
  kind:
    | "system"
    | "developer"
    | "user_profile"
    | "task_memory"
    | "retrieved_doc"
    | "recent_message"
    | "tool_result"
    | "summary";
  text: string;
  tokens: number;
  priority: number;
  createdAt?: string;
};

const priority = {
  system: 100,
  developer: 95,
  user_profile: 85,
  task_memory: 80,
  retrieved_doc: 75,
  tool_result: 70,
  recent_message: 60,
  summary: 55,
};

Then build the final packet by sorting and fitting.

export function fitContext(items: ContextItem[], tokenLimit: number) {
  const sorted = [...items].sort((a, b) => b.priority - a.priority);
  const selected: ContextItem[] = [];
  let used = 0;

  for (const item of sorted) {
    if (used + item.tokens > tokenLimit) continue;
    selected.push(item);
    used += item.tokens;
  }

  return { selected, used, dropped: sorted.length - selected.length };
}

This simple strategy already beats “append everything” because it makes context selection explicit. Later, you can improve it with recency, semantic relevance, account tier, user intent, or safety rules.

Expected result: your agent keeps the important material when context pressure rises. It drops low-value history first instead of randomly losing the thread.

Step 3: Count Tokens Before The API Call

Token counting is not optional. Sending the request and hoping the provider truncates it cleanly is how you get incoherent answers and impossible-to-debug failures.

A rough TypeScript token counter:

import { encodingForModel } from "js-tiktoken";

const enc = encodingForModel("gpt-4o");

export function countTokens(text: string) {
  return enc.encode(text).length;
}

export function countMessageTokens(messages: Array<{ role: string; content: string }>) {
  return messages.reduce((total, message) => {
    return total + countTokens(message.role) + countTokens(message.content) + 4;
  }, 0);
}

The + 4 is a rough message overhead placeholder. Provider message formats differ, so calibrate this against real usage metadata from your API responses. The point is not perfect counting. The point is preventing obvious disasters before they reach the model.

Add a guardrail:

export function assertWithinBudget(inputTokens: number, maxInputTokens: number) {
  if (inputTokens > maxInputTokens) {
    throw new Error(
      `Context budget exceeded: ${inputTokens} input tokens used, limit is ${maxInputTokens}`
    );
  }
}

Expected result: oversized context becomes an application event you can handle, not a model error your user experiences.

Step 4: Split Memory Into Short-Term And Long-Term

Short-term memory is the current thread. Long-term memory is durable knowledge about the user, account, project, or workflow.

Do not mix them.

Bad memory design:

const memory = [
  "User said hello",
  "User likes concise answers",
  "User uploaded Q3 vendor contract",
  "Tool call failed at 14:03",
  "User is on enterprise plan",
  "Assistant apologized",
];

Better memory design:

type ShortTermMemory = {
  threadId: string;
  recentMessages: Message[];
  rollingSummary: string;
  lastToolResults: ToolResult[];
};

type LongTermMemory = {
  userId: string;
  preferences: {
    tone?: "brief" | "detailed";
    timezone?: string;
  };
  accountFacts: {
    plan?: string;
    permissions?: string[];
  };
  durableNotes: string[];
};

Short-term memory changes constantly. Long-term memory should be promoted only when it is durable and useful.

“User asked about refunds” is not long-term memory.

“User manages billing for account Acme-123” might be.

Expected result: your agent remembers what matters across sessions without dragging every conversation artifact into every request.

Step 5: Summarize Old Turns Without Lying To Yourself

Summaries are useful. They are also lossy. Treat them like compressed state, not gospel.

Use a structured summary format:

const summaryPrompt = `
Summarize the older conversation for future agent context.

Return JSON with:
- user_goal: current goal if still active
- decisions: confirmed decisions
- constraints: hard requirements, deadlines, preferences
- open_questions: unresolved questions
- entities: people, accounts, files, tickets, products
- warnings: safety, compliance, or ambiguity notes

Do not include small talk.
Do not invent facts.
If something is uncertain, mark it uncertain.
`;

Store the result like this:

type RollingSummary = {
  threadId: string;
  version: number;
  coveredMessageIds: string[];
  summaryJson: {
    user_goal?: string;
    decisions: string[];
    constraints: string[];
    open_questions: string[];
    entities: string[];
    warnings: string[];
  };
  updatedAt: string;
};

Trigger summarization when recent messages cross a threshold:

if (recentMessageTokens > budget.recentMessagesTokens) {
  const olderMessages = selectOldestMessages(recentMessages);
  const summary = await summarizeMessages(olderMessages);
  await saveRollingSummary(threadId, summary);
  await archiveMessages(olderMessages);
}

Expected result: old context gets compressed into useful state instead of being deleted blindly or stuffed into the prompt forever.

Step 6: Retrieve Context Instead Of Stuffing It

If your agent uses docs, tickets, policies, contracts, code, or CRM data, do not preload everything. Retrieve only what matches the current task.

The retrieval packet should be capped too:

type RetrievedChunk = {
  id: string;
  source: string;
  title: string;
  text: string;
  score: number;
  tokens: number;
};

export function selectRetrievedChunks(chunks: RetrievedChunk[], tokenLimit: number) {
  const sorted = chunks
    .filter((chunk) => chunk.score >= 0.72)
    .sort((a, b) => b.score - a.score);

  const selected: RetrievedChunk[] = [];
  let used = 0;

  for (const chunk of sorted) {
    if (used + chunk.tokens > tokenLimit) continue;
    selected.push(chunk);
    used += chunk.tokens;
  }

  return selected;
}

Add source labels directly into the context:

function formatRetrievedContext(chunks: RetrievedChunk[]) {
  return chunks
    .map((chunk) => {
      return `<source id="${chunk.id}" title="${chunk.title}" url="${chunk.source}">
${chunk.text}
</source>`;
    })
    .join("\n\n");
}

This helps the model separate retrieved evidence from conversation history. It also makes citations and debugging less painful.

Expected result: the agent gets the right documents for the current task without rereading your entire knowledge base like a confused intern with unlimited coffee.

Step 7: Put Stable Context First For Prompt Caching

Prompt caching rewards stable prefixes. Providers differ in the details, but the practical rule is simple:

Put reusable, boring, rarely changing content at the front of the prompt.

Good order:

Tool definitions
System instructions
Safety and compliance rules
Product rules or static operating manual
Dynamic user/account state
Retrieved chunks
Recent messages
Current user request

Bad order:

Timestamp
Random request ID
Current user message
System prompt
Tool definitions
Retrieved docs

That bad order busts caching because the prefix changes every request.

Example packet builder:

export function buildPromptPacket(input: {
  systemPrompt: string;
  staticPolicy: string;
  userState: string;
  retrievedContext: string;
  rollingSummary: string;
  recentMessages: Message[];
  currentRequest: string;
}) {
  return [
    {
      role: "system",
      content: `${input.systemPrompt}\n\n${input.staticPolicy}`,
    },
    {
      role: "developer",
      content: `User/account state:\n${input.userState}`,
    },
    {
      role: "developer",
      content: `Conversation summary:\n${input.rollingSummary}`,
    },
    {
      role: "developer",
      content: `Retrieved context:\n${input.retrievedContext}`,
    },
    ...input.recentMessages,
    {
      role: "user",
      content: input.currentRequest,
    },
  ];
}

Keep timestamps, random IDs, and request-specific metadata out of the stable prefix unless the model truly needs them.

Expected result: repeated traffic can benefit from provider caching, and your latency/cost profile gets less ugly without changing the agent’s behavior.

Step 8: Add Context Observability

You cannot manage what you do not log. Yes, that sentence is dangerously close to business-book territory. It is still true.

Log these per model call:

type ContextLog = {
  requestId: string;
  threadId: string;
  model: string;
  inputTokensEstimated: number;
  inputTokensActual?: number;
  outputTokensActual?: number;
  cachedTokens?: number;
  contextItemsSelected: number;
  contextItemsDropped: number;
  retrievedChunks: number;
  summaryVersion?: number;
  latencyMs: number;
  finishReason?: string;
  error?: string;
};

Then create alerts for:

Input tokens above 85% of your budget
Cache hit rate dropping suddenly
Retrieval chunks consistently hitting the cap
Summaries growing without bound
Tool results consuming more than recent messages
Context overflow errors
Latency increasing with conversation length

Expected result: you catch context rot early. You can see whether the agent is slow because retrieval is bloated, summaries are huge, cache is busted, or users are pasting novels.

Step 9: Test With Mean Conversation Fixtures

Do not test context management with polite three-turn chats. Production users are chaos with keyboards.

Create fixtures like:

const fixtures = [
  "50-turn support thread with repeated plan changes",
  "User pastes a 20,000-token contract and asks one narrow question",
  "Tool returns a massive JSON payload",
  "User changes their goal halfway through",
  "Retrieved docs contain conflicting policy versions",
  "Conversation includes sensitive data that should not become long-term memory",
];

For each fixture, assert:

type ContextTestResult = {
  stayedWithinBudget: boolean;
  preservedSystemPrompt: boolean;
  includedCurrentUserRequest: boolean;
  includedRelevantRetrievedDocs: boolean;
  excludedIrrelevantOldMessages: boolean;
  answerWasCoherent: boolean;
};

Expected result: you stop discovering context bugs from angry users. Much cheaper.

Common Pitfalls

Pitfall 1: Trusting Automatic Truncation

Automatic truncation is a seatbelt, not a driving strategy. It can drop early conversation items that still matter, break cache locality, or remove setup details the model needed.

Fix it: make truncation explicit in your own context builder. If you enable provider truncation, treat it as a last-resort backup.

Pitfall 2: Keeping Tool Results Forever

Tool outputs are often huge. Search results, database rows, logs, and JSON payloads can devour context.

Fix it: summarize tool results into durable facts. Store raw outputs outside the prompt and retrieve them only when needed.

Pitfall 3: Summarizing Away The Important Bit

A bad summary can delete the user’s actual constraint and keep the fluff. Congratulations, you compressed the wrong thing.

Fix it: use structured summaries with fields for decisions, constraints, open questions, and warnings. Keep references to covered message IDs so you can audit what got compressed.

Pitfall 4: Polluting The Cache Prefix

If your stable prompt starts with Current time: 2026-08-03T14:03:22Z, every request has a different prefix. Cache gone.

Fix it: put dynamic metadata later in the prompt. Keep the front stable.

Pitfall 5: Treating RAG As A Bigger Context Window

Retrieval is not permission to stuff 40 chunks into every request. Bad RAG is just context bloat with embeddings.

Fix it: cap retrieved tokens, deduplicate chunks, filter by score, and prefer fewer high-quality sources.

Pitfall 6: Saving Sensitive Data Into Memory

Agents love remembering things. Compliance teams love when they do not remember passwords, tokens, private keys, medical details, and random personal data forever.

Fix it: run memory writes through a filter. Block secrets and sensitive data from long-term memory by default.

A Production Context Policy You Can Steal

Use this as your baseline:

export const contextPolicy = {
  maxBudgetUsage: 0.8,
  alwaysInclude: ["system", "developer", "current_user_request"],
  includeWhenRelevant: ["user_profile", "task_memory", "retrieved_doc", "tool_result"],
  compressWhenOld: ["recent_message", "tool_result"],
  neverStoreLongTerm: ["password", "api_key", "access_token", "credit_card"],
  retrieval: {
    maxChunks: 8,
    minScore: 0.72,
    maxTokens: 30000,
  },
  summaries: {
    triggerTokens: 20000,
    format: "structured_json",
    keepRecentTurns: 12,
  },
};

This is not sacred. Tune it. But start with a policy, not a pile of conditional string concatenation.

The Real Takeaway

LLM context window management in production is not about cramming more tokens into the model. That is the expensive beginner instinct.

The grown-up move is control:

Budget the prompt before the request
Prioritize context by value
Summarize old conversation state
Retrieve only relevant external data
Preserve stable prefixes for caching
Log the context packet like real infrastructure

A production agent should know what to remember, what to retrieve, what to compress, and what to ignore. Build that layer now, before your agent becomes slow, pricey, and weird in front of actual users.