RAG Query-Rewriting Evaluation: Prove the Rewrite Finds Better Evidence

RAG systems often rewrite a user’s message before search. A conversational question such as “does it cover that overseas?” may become a standalone query containing the product, policy, and country inferred from prior turns.

The rewrite can improve recall. It can also invent a country, erase negation, replace the user’s exact identifier, or over-specify an interpretation that excludes the right document. Fluency is not evidence of retrieval quality.

Evaluate query rewriting as an information-retrieval component with observable inputs, outputs, and judged documents.

Define the rewrite’s job

Different systems need different transformations:

• resolve references from conversation history;
• expand acronyms and aliases;
• decompose a multi-part question;
• translate into the corpus language;
• add structured filters;
• generate lexical and semantic variants.

Name the mode in the request and log it. A rewrite intended to resolve pronouns should not silently add likely answers.

Create an explicit contract:

{
  "standaloneQuery": "...",
  "filters": {"product": "...", "region": "..."},
  "preservedTerms": ["quoted identifier"],
  "ambiguities": []
}

If a required reference cannot be resolved, returning an ambiguity is safer than choosing the most probable entity.

Build judgments around evidence

Collect real query sessions and annotate which documents or passages can answer them. Preserve the conversation turns required for reference resolution.

Include hard slices:

• short follow-ups such as “what about Germany?”;
• negation and exclusions;
• product codes, names, and error strings;
• multilingual queries;
• time-sensitive policies;
• questions with two legitimate interpretations;
• multi-hop questions needing several documents.

An annotation can contain relevant, partially relevant, and harmful documents. Harmful judgments are useful when an old or similarly named policy would lead to a wrong answer.

Compare against strong baselines

Always measure the raw user query. Add deterministic baselines such as conversation-title prefixing, acronym dictionaries, and exact-entity preservation. A complex model rewrite must beat these options enough to justify cost and latency.

For each configuration, run the same retriever and index. Report recall@k, mean reciprocal rank, normalized discounted cumulative gain, and the rate at which a harmful document enters top-k.

Do not change the embedding model, reranker, and rewrite prompt in one experiment. You will not know which component produced the delta.

Test preservation invariants

Before retrieval, compare the rewrite with the source request. Quoted strings, numbers, units, negation, dates, and explicit entities should be preserved unless a documented normalization rule applies.

assertSame(extractQuoted(raw), extractQuoted(rewrite));
assertSame(extractNumbers(raw), extractNumbers(rewrite));
assertSame(extractNegation(raw), extractNegation(rewrite));

When conversation history supplies an entity, record the source turn. This makes it possible to distinguish correct resolution from model invention.

Reject rewrites that contain instructions, answers, or unverifiable filters. Search queries should describe the evidence need, not persuade the retriever.

Evaluate decomposition separately

A multi-part question may benefit from several focused queries. Score whether every required facet produced at least one relevant document, not whether any one subquery did well.

Control query explosion. Ten rewrites can improve recall while multiplying search cost and filling the candidate pool with noise. Measure unique relevant documents per query and marginal gain from each additional branch.

Merge results with a documented rank-fusion method, then deduplicate by document lineage. Otherwise repeated variants can make one source look independently supported.

Measure final answer grounding

Better retrieval metrics should improve grounded answers, but the generator may fail to use the evidence. Run an end-to-end comparison after component evaluation.

Score factual support, completeness by facet, citation correctness, abstention when evidence is missing, and contradiction with authoritative sources. Pair the rewritten and raw-query answers on the same examples.

Keep retrieval and generation diagnoses separate. If the correct passage entered context but the answer ignored it, changing the rewrite prompt is the wrong fix.

Shadow before enabling

In production shadow mode, run the rewrite and retrieve candidates without changing the user-visible answer. Compare the candidate set with the current system and inspect cases with large rank changes.

Log latency, token use, query count, filters, top document IDs, and index revision. Redact user data while preserving stable test features.

Enable the rewrite for a sticky cohort, then monitor search-empty rate, corrections, citation failures, and escalation. Add automatic fallback to the raw query when invariants fail or the rewrite produces no results.

Watch for corpus drift

A rewrite tuned to one index can degrade when document names, taxonomy, or language distribution changes. Version the rewrite policy with the corpus and rerun benchmark slices after major ingestion changes.

Monitor entity-resolution failures and the vocabulary distance between queries and newly added content. Deterministic alias maps may need updating even when the prompt does not.

Promote difficult production examples into the evaluation set, including the original history, judged evidence, and expected ambiguity behavior.

The takeaway

A query rewrite is good only when it retrieves better evidence without changing the user’s intent. Test it against annotated documents, preservation invariants, raw-query baselines, and end-to-end grounded answers. Shadow large rank changes and keep a fallback. Never ship a rewrite because it reads well in a prompt playground.