Changing an embedding model can look like a routine dependency upgrade. Replace the model identifier, deploy the service, and continue querying the existing vector index. That approach can silently damage retrieval.

An embedding is meaningful relative to the representation space produced by its model. If stored document vectors came from one model while new query vectors come from another, their coordinates generally do not have a shared meaning. Matching dimensions are not enough to make the vectors compatible.

This article explains the mental model behind that problem, then develops a safe migration pattern: version the embedding space, build a new index beside the old one, evaluate both on the same queries, and switch traffic only after the new system meets the required quality and operational targets.

Treat an embedding model as part of the index schema

A vector index does not contain model-independent meanings. It contains numeric coordinates produced by a particular embedding process.

Suppose the current system uses model A:

documents -> model A -> vectors A -> index A
queries   -> model A -> vectors A -> search index A

Both sides use the same representation function, so a similarity measure such as cosine similarity compares vectors in one shared space.

Now imagine changing only the query path:

documents -> model A -> vectors A -> index A
queries   -> model B -> vectors B -> search index A

Even if models A and B both emit 768-dimensional vectors, coordinate 42 in one space does not have to represent anything comparable to coordinate 42 in the other. Two embedding models can rotate, rescale, reorganize, or otherwise learn different representations of the same inputs.

The practical rule is therefore:

query embedding version == indexed document embedding version

unless the models were explicitly designed and validated to produce compatible representations. Do not infer compatibility from vector dimension, architecture family, provider, or similar model names.

Thinking of the embedding model as part of the index schema makes upgrades easier to reason about. Changing the schema requires migrating the stored representation, not merely changing the code that creates new queries.

See the failure with a small example

Consider a toy two-dimensional space. Model A places three documents and a query like this:

                 x      y
query A        0.9    0.1
doc cats       0.8    0.2
doc billing   -0.7    0.1
doc travel     0.0    0.9

The query is close to doc cats, which is what we want.

Suppose model B represents the same query as:

query B        0.1    0.9

Searching model A’s stored vectors with that model B query can now make doc travel look closest. Nothing is wrong with either model. The mistake is comparing coordinates from different spaces as if they shared an axis system.

Real embeddings have many more dimensions and their geometry is not this easy to inspect, but the compatibility problem is the same.

There is another important consequence: if a migration changes preprocessing as well as the model, that preprocessing belongs to the representation version too. Changing truncation, prefixes, normalization, chunk text, or which fields are embedded can alter retrieval even when the model itself stays fixed.

Give every representation an explicit version

Before building a replacement index, define what identifies an embedding space. A useful version record can include:

embedding_version: support-v3
model:            <exact model identifier>
preprocessing:    <versioned pipeline>
dimensions:       <output dimension>
metric:           cosine
chunking:         <versioned chunk policy>

The exact metadata depends on the system. The important point is that embedding_version refers to the complete representation contract rather than a vague label such as latest.

Store that version with indexed records or make it an immutable property of the index. Also include it in logs for retrieval requests. When quality changes later, you should be able to answer which model and preprocessing produced both the query vector and the candidate vectors.

Avoid silently reusing an index name for a different representation. Names such as these make operations easier to audit:

support-chunks-emb-v2
support-chunks-emb-v3

A logical alias such as support-chunks-active can point to whichever physical index currently serves production traffic.

Build the new index beside the old one

The safest general migration is a side-by-side rebuild.

                     -> model A -> index A -> current production
source documents ----|
                     -> model B -> index B -> candidate system

Re-embed the source corpus with the new representation pipeline and write those vectors to a separate index. Keep the old index intact while this happens.

This separation has several advantages. Production queries continue using a known representation. Partial backfills cannot mix vector spaces. Evaluation can compare complete systems. Rollback remains possible because the old index has not been overwritten.

Keep source data independent from vector storage

A migration is much easier if the vector index is derived state rather than the only copy of the content. Keep stable document or chunk identifiers and enough source data to regenerate embeddings.

A rebuild then becomes conceptually simple:

for each source chunk:
    text = preprocess(chunk, version="v3")
    vector = embed(text, model="model-b")
    write(index="support-chunks-emb-v3", id=chunk.id, vector=vector)

This is pseudo-code, not a provider-specific API. In production, add batching, retry handling, rate limits, idempotent writes, progress checkpoints, and validation of failed records as appropriate for the embedding service and index.

Stable identifiers matter because they let you compare the same content across old and new indexes. They also make incremental repair possible without guessing which vector corresponds to which source record.

Handle writes that occur during a long rebuild

If the corpus changes while backfilling, a snapshot taken at the start can become stale before the new index is ready.

Common approaches include:

  • rebuild from a consistent source snapshot, then replay changes that happened after the snapshot;
  • dual-write new or updated content to both representation pipelines during the migration window;
  • run a final incremental synchronization before the cutover.

The right choice depends on write volume and infrastructure. A mostly static documentation corpus may need only a short final synchronization. A rapidly changing product catalog may require explicit change capture or dual writes.

Whichever approach you use, define a completion condition. “Most vectors were written” is not enough if the missing records contain important content.

Evaluate retrieval before evaluating generation

For a RAG application, it is tempting to compare only final model answers. That can hide retrieval regressions because generation adds another source of variation.

First compare the retrieval systems directly on a fixed evaluation set:

same query set
    |-> model A + index A -> ranked results A
    |-> model B + index B -> ranked results B

Each evaluation query should have a notion of useful evidence. Depending on the application, that may be a relevant document, one or more relevant chunks, or a set of acceptable sources.

Useful retrieval measurements can include recall at a chosen cutoff, precision at a chosen cutoff, reciprocal rank, or another metric that matches the product’s retrieval requirement. The metric name matters less than connecting it to the task. If the generator needs at least one supporting passage in the first five results, measure whether that evidence appears there.

Do not judge an embedding migration from a few hand-picked queries. Include ordinary cases and important slices such as:

  • short and long queries;
  • exact product names or identifiers;
  • paraphrases that share little wording with the source;
  • ambiguous queries;
  • rare topics;
  • languages or domains the product actually serves.

A single aggregate score can hide a severe regression in a small but important slice.

After retrieval quality is acceptable, run end-to-end RAG evaluation. The new ranking may change which passages enter the context, their order, and how much relevant evidence fits within the context budget. Those changes can affect answer quality even when retrieval metrics improve.

Compare quality, latency, and cost together

A new embedding model is not an upgrade merely because it scores better on one retrieval benchmark.

Indexing cost

Re-embedding a large corpus consumes embedding inference and index-write capacity. If the corpus contains N chunks, a full migration requires generating roughly N new document embeddings, plus any retries or changed records. Estimate this work before starting and throttle it so that the backfill does not interfere with production workloads.

Query latency

Measure query embedding latency separately from vector search latency. A model that improves retrieval but takes substantially longer to embed a query can hurt an interactive product. Conversely, a smaller or locally served model may reduce embedding latency while changing retrieval quality.

Storage and memory

A change in vector dimension changes vector payload size. For the same numeric representation, doubling the dimensions roughly doubles the raw vector payload, but it does not imply that the entire index doubles because indexes also contain metadata and search structures. Measure the actual index rather than extrapolating total storage from vector dimensions alone.

Downstream token cost

Better ranking can sometimes let an application use fewer retrieved chunks, but that is an empirical product decision rather than a property guaranteed by the embedding model. Evaluate answer quality at the context sizes you intend to deploy.

The migration decision should use constraints that matter in production, not just a leaderboard score.

Shadow traffic before the cutover

An offline evaluation set is necessary but rarely captures every production query pattern. Shadow evaluation can expose differences on real traffic without changing user-visible results.

For a sample of eligible queries:

production: query -> model A -> index A -> returned results
shadow:     query -> model B -> index B -> logged results only

Compare retrieval metrics when labels are available, and otherwise inspect measurable signals such as result overlap, score distributions, latency, empty-result rates, and selected query slices. Low result overlap is not automatically bad: the new system is supposed to change some rankings. Treat it as a reason to investigate, not as a quality metric by itself.

Be careful with sensitive queries. Shadowing creates another processing and logging path, so existing privacy, retention, and access controls still apply.

Cut over atomically and keep rollback simple

Once the new index is complete and the candidate system passes evaluation, switch query embedding and index selection as one logical change.

Bad cutover:

1. deploy model B for queries
2. later point search to index B

Between those steps, model B queries hit model A vectors.

Safer cutover:

retrieval version v2 -> model A + index A
retrieval version v3 -> model B + index B

active version: v2 -> v3

The application chooses a complete retrieval version, and each version binds the query model to its matching index.

Keep the previous version available for a rollback window if operational constraints permit. A rollback should restore both the old query embedding path and the old index together.

If the new model changes vector dimensions, a separate index is usually unavoidable because an existing vector field commonly has a fixed dimension. Even when dimensions match, separation is still valuable because it prevents accidental mixing and makes rollback straightforward.

Avoid common migration mistakes

Checking only vector dimensions. Equal dimensions make two vectors structurally comparable to software, not semantically compatible. Require an explicit representation version.

Updating documents lazily in one shared index. If some stored vectors come from model A and others from model B, one query vector is being compared across mixed spaces. Use separate indexes or another design that guarantees searches stay within one representation version.

Changing several retrieval components without tracking them. A new model, new chunking policy, new similarity metric, and new reranker in one release make regressions difficult to diagnose. When changes must ship together, version the whole pipeline and evaluate the combined system; otherwise isolate changes when practical.

Comparing raw similarity scores across models. Similarity-score distributions can change between embedding models. A cosine score of 0.75 from one model is not guaranteed to carry the same relevance meaning for another. Revalidate any thresholds that depend on scores.

Deleting the old index immediately. A successful offline test does not rule out production regressions. Retain a rollback path long enough to observe the new system under representative traffic when storage and policy allow it.

Ignoring newly written content during backfill. A perfectly rebuilt snapshot can still be incomplete at cutover. Reconcile changes made during the migration window.

Know when a full migration is unnecessary

Not every embedding-related change requires rebuilding every vector.

If you change only the vector search implementation while preserving the exact stored vectors and their scoring semantics, you may be able to copy or rebuild the index structure without re-embedding source text.

If you change only a downstream reranker that operates on retrieved candidates, the embedding space itself has not changed. Evaluate the reranker, but do not regenerate vectors merely because another retrieval stage changed.

If you add metadata filters without changing embedded text or vectors, a data migration may be required for metadata but not for embeddings.

The deciding question is simple: did the function that maps source content into vector coordinates change? If yes, assume the stored vectors need a matching migration unless compatibility is an explicit property of the new representation and has been validated for your use case.

Use a migration checklist that protects the invariant

A practical sequence is:

  1. Freeze an explicit definition of the old and new representation versions.
  2. Keep model, preprocessing, dimensions, metric, and chunking metadata auditable.
  3. Build a separate new index from authoritative source content.
  4. Reconcile writes that occur during the backfill.
  5. Validate record counts and failed embedding jobs.
  6. Compare old and new retrieval on the same labeled evaluation set and important slices.
  7. Measure embedding latency, search latency, storage, and migration cost.
  8. Run end-to-end evaluation for applications such as RAG.
  9. Shadow representative traffic when appropriate.
  10. Switch the query model and matching index atomically.
  11. Monitor the new version and preserve a tested rollback path for an appropriate window.

The checklist exists to protect one central invariant: a search request should compare vectors that belong to the same intended representation space.

Conclusion

An embedding model upgrade is a data migration, not just a model configuration change. Stored document vectors encode the geometry of the model and preprocessing pipeline that created them, so new query vectors should not be mixed with an old index merely because their dimensions match.

Version the representation contract, rebuild beside the current index, evaluate retrieval before downstream generation, account for writes during backfill, and cut over the query model and index together. This approach costs temporary storage and migration work, but it makes quality changes measurable and gives production systems a clear rollback path.

Artificial Intelligence Embeddings Vector Search RAG