Retrieval Corpus Drift: Monitor the Knowledge Base Behind Your RAG System

A retrieval-augmented generation system can get worse even when no one changes the model, prompt, or application code. The knowledge base changes underneath it. Policies are replaced, product pages move, duplicate documents accumulate, parsers stop recognizing a new template, and access-control metadata quietly disappears.

This is retrieval corpus drift: a meaningful change in the documents, chunks, metadata, or index that alters what the system can retrieve. Model monitoring alone will not catch it. You need observability for the knowledge pipeline itself.

Define the Corpus You Expect

Start with an inventory contract for every source. Record the owner, connector, expected document count or range, update frequency, supported file types, required metadata, and freshness objective. A policy collection might require a current version for every active region. A support collection might expect successful synchronization every hour.

Do not make exact row counts the only control. Legitimate publishing changes counts. Use several invariants:

• every active product has at least one current support document;
• every chunk has a source ID, version, timestamp, and access label;
• no document has more than one active canonical version;
• the age of the newest successful sync stays inside its objective;
• parser and embedding versions are recorded for every indexed chunk.

Keep a manifest for each ingestion run. It should summarize documents discovered, accepted, rejected, unchanged, deleted, and embedded. Store hashes for raw files and normalized text so you can distinguish a source edit from a parser change.

Measure Coverage, Freshness, and Shape

Corpus monitoring needs more than pipeline success. A connector can return HTTP 200 and ingest an empty folder. Track coverage against the expected entity set, not just the number of files processed.

Freshness has at least three clocks: when the source changed, when the connector observed it, and when the new chunks became searchable. Their differences reveal whether delay comes from polling, parsing, embedding, or index publication.

Watch distributional shape as well. Useful metrics include chunk count per document, token length percentiles, language mix, MIME type mix, metadata null rates, duplicate-hash rate, and the proportion of chunks from each source. A sudden collapse in median chunk length may signal a parser that is extracting navigation fragments. A jump in duplicates may mean versioned files are no longer being canonicalized.

Set alerts on changes that matter, but avoid brittle thresholds. Compare with recent baselines and known publishing events. A planned migration should create a controlled, explained change. An unexplained 40% loss of PDF text at 2 a.m. should stop index promotion.

Test Retrieval With Stable Questions

Structural metrics cannot prove that users can retrieve the right evidence. Maintain a small, versioned evaluation set of questions tied to expected source documents. Include common tasks, high-risk policies, recently changed facts, and known confusions between similar products or regions.

For each question, evaluate whether the expected document appears in the top results, whether forbidden documents stay excluded, and whether the retrieved passage actually contains the evidence. Measure recall at a chosen cutoff, rank, and metadata-filter correctness. Save the query, index version, embedding model, retrieved IDs, and scores so regressions can be reproduced.

Add sentinel facts that intentionally change on a schedule. If a controlled document says the current support code is ORANGE-17, a test should fail after the document is updated to BLUE-42 but the index still returns the old value. This exercises the complete freshness path.

Evaluation sets also drift. Review them when product coverage changes, and keep historical versions so improvement is not manufactured by deleting hard cases.

Gate Index Promotion and Support Rollback

Build a candidate index separately from the serving index. Run manifest checks, metadata policy tests, duplicate analysis, retrieval evaluations, and a small sample of end-to-end grounded answers. Promote only after the candidate passes.

Use immutable index versions or namespaces. If retrieval quality drops, traffic can return to the previous known-good version while ingestion is repaired. Record the source manifest, parser commit, chunking configuration, embedding model, and evaluation result with each version.

Deletion needs equal discipline. Privacy or legal removal requests may require fast erasure, while ordinary source disappearance may be a connector fault. Separate confirmed tombstones from unexplained missing documents so rollback does not resurrect data that must stay deleted.

Make Drift Actionable

An alert should name the affected source, first failing run, changed metric, likely stage, and current serving index. Link to example documents and a comparison with the last good manifest. “RAG quality down” is not actionable; “German policy PDFs parsed to 12% of normal text after parser v18” is.

Assign ownership before incidents. Source teams own publication and canonical versions. Platform teams own ingestion and indexing. Application teams own query behavior and answer evaluation. Security teams own access labels and deletion controls.

RAG systems fail as data systems before they fail as language models. Monitor the corpus as a production dependency, version every transformation, and test retrieval against expected evidence. If you cannot explain what entered the index and what changed, you do not have grounded generation—you have a search-shaped guess.