RAG Retrieval Diversity: Stop Near-Duplicate Chunks From Crowding Out the Answer

Vector search ranks chunks independently. If a policy paragraph appears in a handbook, onboarding guide, FAQ, and archived release note, those near-duplicates can occupy the entire top five. The model sees repetition as confidence while the one exception, definition, or current amendment never enters context.

Retrieval diversity is the discipline of selecting evidence that is relevant and adds information. It is not random variety. The goal is to reduce redundant context without suppressing legitimate corroboration.

Diagnose where duplication enters

Duplicates may be exact copies, overlapping chunks from one document, templated pages that differ only by product name, translated mirrors, or multiple versions of the same policy.

Record parent document, section, version, canonical URL, and content digest during ingestion. Without provenance, the retriever cannot tell whether two similar chunks are independent evidence or the same sentence copied twice.

Start with simple metrics:

• unique parent documents in top-k;
• average pairwise similarity among selected chunks;
• duplicate digest rate;
• coverage of required evidence facets;
• proportion of tokens from the dominant source.

Inspect these by query class. A legal definition may correctly come from one authoritative document, while a comparison question needs several sources.

Remove exact and structural duplicates first

Hash normalized content to eliminate byte-level and whitespace variants. Canonicalize URLs and preserve a redirect or alias table. Detect boilerplate such as navigation, disclaimers, and repeated footers before chunking.

For versioned documents, do not delete history blindly. Mark the current revision and make retrieval policy explicit. Most user questions should search current material; audit requests may require historical versions.

Overlapping chunks need special treatment. If two candidates share most token spans from the same parent, keep the higher-scoring one unless the second contains a distinct required section.

Retrieve broadly, then select

Do not apply diversity to a pool of only k items. Retrieve a larger candidate set—perhaps 20 or 50—then rerank and choose the final context.

Hybrid retrieval can improve the pool. Dense embeddings capture semantic similarity, while lexical search protects exact product names, identifiers, and rare terms. Reciprocal rank fusion combines ranks without requiring scores from different systems to share one scale.

The candidate stage maximizes recall. The selection stage balances relevance, authority, freshness, and novelty.

Use maximal marginal relevance

Maximal marginal relevance, or MMR, selects a candidate using both similarity to the query and dissimilarity from items already selected:

score = λ × relevance(query, candidate)
      - (1 - λ) × max_similarity(candidate, selected)

A high lambda favors pure relevance. A lower value favors novelty. Tune it by query type rather than adopting one global default.

MMR works best after hard filters for tenant, permissions, date, language, and document status. Diversity must never pull an unauthorized or obsolete chunk into context merely because it differs.

Add provenance-aware constraints

Similarity alone cannot recognize editorial independence. Add constraints such as no more than two chunks per parent document, prefer distinct sections, and select at least one authoritative source when available.

For multi-part questions, identify facets before selection. A query about pricing, cancellation, and data retention needs evidence for all three. Allocate slots by facet, then rerank within each group.

const requiredFacets = ["price", "cancellation", "retention"];
const selected = requiredFacets.flatMap(facet => bestForFacet(facet, candidates, 1));

Facet coverage can outperform generic diversity because it connects selection directly to answer requirements.

Preserve corroboration deliberately

Not every repeated claim is waste. Independent sources can increase confidence, and multiple passages may establish scope or an exception.

Track source lineage. Two pages generated from the same database are one evidence family even if their wording differs. Two independent primary records can be legitimate corroboration.

Allow the final context to include more than one source for high-stakes claims, but make that a policy rule rather than an accident of embedding similarity.

Evaluate answer impact

Retrieval metrics are necessary but insufficient. Build questions with annotated evidence facets and known duplicate traps. Measure recall of required evidence, unique-source coverage, stale-source rate, and answer correctness with citations.

Compare four configurations: baseline top-k, exact deduplication, MMR, and provenance-aware facet selection. Track context tokens and latency alongside quality.

Include adversarial cases where ten near-identical outdated pages outrank one current policy. The correct system should prefer freshness and authority before diversity.

Observe production collapse

Log the selected chunk IDs, parent IDs, revisions, ranks, and pairwise similarity summary. Alert when one parent consumes most context or when duplicate rate rises after an ingestion change.

User corrections such as “you ignored the exception” are valuable labels. Trace them back to whether the evidence was absent, present but unused, or contradicted.

Cache final retrieval only with corpus and selection-policy versions. Otherwise a fixed cache can conceal improvements to deduplication.

The takeaway

Top-k is a ranking, not a context strategy. Retrieve a broad authorized pool, remove exact and overlapping duplicates, then select for relevance, freshness, authority, and incremental information. Measure whether the final context covers the question’s facets. Diversity is useful only when it helps the model see evidence it would otherwise miss.