Embedding Model Migration: Change Vectors Without Breaking Search

An embedding upgrade looks like a model-name change until retrieval quality collapses. New query vectors are compared with old document vectors, dimensions no longer match, or the same cosine threshold describes a completely different score distribution.

Embeddings are coordinates in a model-specific space. Migration requires a new index contract, not an in-place model swap.

Version every part of the representation

Define an embedding version that includes provider, model, output dimension, normalization, distance metric, preprocessing, chunking policy, and any instruction prefix. Two vectors from different contracts should never share a searchable namespace.

Store the version beside each vector and in index metadata. Keep source document IDs and chunk IDs stable so old and new representations can be compared. Preserve the source text or a durable pointer; without it, re-embedding becomes a data-recovery project.

Do not assume larger dimensions mean better retrieval. Evaluate the new model on the language, document type, query length, and domain vocabulary your users actually produce. Also measure latency, storage, indexing throughput, and cost.

Build the new index beside the old one

Backfill a separate index with the new contract. Process records in restartable batches, checkpoint progress, and record failed documents without stopping the entire migration.

While backfill runs, dual-write new and changed content to both versions. Make each write idempotent using document version and chunk ID. Deletions also need dual propagation; otherwise the new index can resurrect content removed from the old system.

If preprocessing changed, compare chunks as well as vectors. A new model evaluated on improved chunks is a valid product candidate, but the experiment should say clearly which layer caused the gain.

Verify counts, tenant partitions, access-control metadata, deletion tombstones, and a sample of source-to-vector lineage before sending any user traffic.

Shadow queries before routing answers

Run production queries against both indexes while continuing to serve results from the old one. Log ranked document IDs, scores, latency, filter behavior, and downstream answer quality using privacy-safe identifiers.

Overlap metrics such as recall at k and rank correlation reveal how much retrieval changed, but disagreement is not automatically bad. Create a judged set of real questions with relevant documents and failure labels. Include rare queries, multilingual inputs, exact identifiers, and adversarial near-duplicates.

Recalibrate score thresholds. Cosine similarity from one model is not interchangeable with cosine similarity from another. If the application uses a cutoff to decide “no answer,” fit it again using labeled examples.

For RAG, evaluate the final answer too. A new retriever may return stronger evidence but change ordering in a way that interacts with context limits, rerankers, or citation extraction.

Cut over gradually and keep rollback cheap

Route a small percentage of traffic to the new index, segmented by tenant or stable request hash. Monitor retrieval success, empty-result rate, answer quality, latency, cost, and support signals. Avoid per-request random switching when users expect repeated queries to behave consistently.

Keep the old index read-only but available through a defined rollback window. Freeze writes only after dual-write lag reaches zero and reconciliation passes. Then switch the default version in configuration rather than deleting infrastructure during the same change.

Retire the old index after retention, audit, and rollback requirements are satisfied. Record the migration decision and evaluation set so the next upgrade can compare against more than memory.

An embedding migration succeeds when the new geometry is isolated, measured, and reversible. The safest upgrade is not the cleverest model; it is the one whose retrieval behavior you can explain before every user depends on it.