Context Compaction Regression Testing for Long-Running AI Agents

Context compaction is the moment your long-running agent rewrites its own history. That can cut token cost dramatically. It can also delete the user’s veto, merge two similarly named files, or turn an unresolved assumption into a confirmed fact.

Do not grade a summary by how readable it sounds. Grade it by whether future actions remain correct. Compaction is a state transformation, so test it like one.

What Must Survive Compaction?

Define typed invariants before collecting examples. Common classes include immutable user constraints, approved scope, forbidden actions, file and account identifiers, unresolved questions, completed effects, deadlines, and provenance for remembered facts.

Represent critical state outside prose when possible. A summary can explain the work, but a structured ledger should hold approvals, effect IDs, and task status. Your test compares both the compacted text and the machine-readable state.

{
  "must_preserve": ["never delete source files", "deploy only staging"],
  "open_questions": ["which tenant owns dataset B?"],
  "completed_effects": ["invoice:inv_204 sent"]
}

How Do You Build a Useful Test Corpus?

Start with real sanitized traces, then add adversarial histories. Put an important constraint early and contradict it later in quoted text. Use two people with similar names. Include a plan that was replaced, a tool call that timed out after succeeding, and a user correction delivered near the compaction boundary.

Test history	Expected survival
Old plan replaced by new plan	New plan only, replacement noted
Ambiguous tool timeout	Effect marked unknown, not failed
User says “do not publish”	Publication remains forbidden
Two similar file paths	Exact target path preserved

Generate histories at several lengths. Failures often appear only when the model is under real compression pressure.

What Should the Evaluator Measure?

Use deterministic checks for exact identifiers, enums, dates, negations, and counts. Add an LLM judge for semantic preservation, but require citations back to source turns so fluent invention is penalized.

The strongest test is downstream replay. Give a fresh agent only the compacted state and ask it to continue the task. Score whether it chooses the right next action, respects constraints, avoids repeating completed effects, and asks about genuine unknowns.

Track false deletion and false invention separately. Missing a minor detail is not the same as inventing approval for a destructive action.

How Do You Gate a New Compactor?

Run the old and new compactors on the same frozen corpus. Compare invariant retention, downstream success, token reduction, latency, and cost. Set hard zero-tolerance gates for safety constraints and effect identities; use statistical thresholds for softer narrative details.

Canary the new version on low-risk tasks, log the source span supporting each retained fact, and keep the previous compacted state for rollback. When incidents occur, add the full pre-compaction trace to the regression corpus after sanitization.

A compactor is not merely a summarizer. It is a migration function for an agent’s operational state. If you would not deploy a database migration without tests, do not deploy this one on vibes.