A single embedding compresses an entire query or document into one vector before similarity is computed. That representation is convenient for approximate nearest-neighbor search, but every token-level signal must survive the compression step. Late interaction retrieval keeps the independent encoding property while postponing part of the query-document comparison until search time.
The core change is representational. Instead of storing one vector per document, a late interaction model can retain a set of contextual token vectors. A query is also represented by multiple vectors. Relevance is then computed from interactions between those two sets rather than from one global dot product.
This creates a middle ground between single-vector retrieval and cross-encoder scoring: document representations can be precomputed, yet query tokens can still match document tokens individually.
Independent encoding keeps documents reusable
A cross-encoder processes a query and document together. Its attention layers can model detailed interactions between the pair, but the resulting computation depends on both inputs. A document cannot be fully encoded once and reused for arbitrary future queries.
Late interaction separates the expensive contextual encoding. The document encoder produces token-level vectors without seeing the eventual query, so those vectors can be computed before requests arrive. The query encoder does the same for the incoming query.
Only the interaction stage depends on the pair. In the ColBERT formulation, that stage is deliberately simple enough to operate over precomputed document vectors.
This separation matters operationally. It moves transformer encoding of the corpus out of the request path, but it does not reduce each document to a single coordinate in embedding space.
MaxSim keeps the strongest token match
A common late interaction score uses a maximum-similarity operation for each query vector. For query vectors q_i and document vectors d_j, the score can be expressed conceptually as
score(Q, D) = sum_i max_j similarity(q_i, d_j)Each query token representation finds its strongest matching document token representation. Those maxima are then summed across the query.
The operation has a specific consequence: different query tokens can select different document positions. A document does not need one global vector that simultaneously represents every aspect of the query.
Contextual encoding still matters before this comparison. A token vector reflects its surrounding text, so identical surface tokens can receive different representations in different contexts. The interaction is token-granular, but it is not merely lexical matching.
MaxSim also discards information. For each query vector, all document similarities except the maximum vanish from the final sum. Repeated moderate matches do not accumulate in the same manner as a sum over every token pair. The aggregation rule therefore defines a particular relevance signal rather than a generic measure of semantic relatedness.
Multi-vector indexes exchange storage for interaction detail
Retaining multiple vectors per document increases index volume compared with one-vector dense retrieval. If a document contributes dozens or hundreds of token vectors, storing every vector at full precision can dominate the retrieval footprint.
Compression and pruning can reduce that cost, but they alter the engineering problem. The index now needs to support candidate discovery across many document vectors and then recover document-level scores from token-level matches.
This is not equivalent to placing one document embedding in a standard vector index. A naive search over every stored token vector can return many hits from the same document and perform substantial work before a final document ranking is available.
Practical late interaction systems therefore combine vector indexing, compression, pruning, or staged scoring. The exact retrieval plan is implementation-specific, while the architectural invariant remains the same: document-side contextual vectors are available before the query, and fine-grained interaction occurs later.
Token count changes both storage and scoring cost
Document length has a direct systems effect in a multi-vector representation. More retained document tokens mean more vectors to store and more potential matches to consider. Query length similarly changes the number of MaxSim terms in the score.
Token filtering can reduce this footprint, but removing vectors is not neutral. A discarded token can no longer provide the strongest match for any query vector. The value of pruning therefore depends on which token representations carry retrieval signal for future queries.
Chunking introduces another boundary. If a long document is split before encoding, contextual token vectors are conditioned only on their chunk context. Retrieval then ranks chunks or aggregates them back to documents according to the application design.
These choices affect representation semantics as well as index size. A smaller chunk can isolate a relevant passage, while a larger chunk supplies broader context to each token representation. Neither effect follows from late interaction alone.
Similarity normalization belongs to the model definition
The interaction score depends on the geometry produced by the encoders and on the similarity function expected during retrieval. Dot product and cosine similarity are equivalent only when the compared vectors have the required normalization.
An implementation should not insert vector normalization merely because its search library defaults to cosine distance. If the trained model expects normalized token vectors, that normalization is part of the representation pipeline. If it expects another scoring convention, changing the convention changes rankings.
The same applies to quantization. Compressing document vectors can alter individual similarities, and MaxSim can amplify ranking changes near a maximum boundary. A small perturbation that swaps the top two token matches changes which value contributes to the score.
Index compression should therefore be evaluated on retrieval behavior, not only average vector reconstruction error.
Late interaction and reranking occupy different request paths
A cross-encoder reranker typically receives a limited candidate set from an earlier retrieval stage. It can spend substantial computation on each query-document pair because the candidate count has already been reduced.
Late interaction can also be used as a reranker, but its independent document encoding allows a different option: the same representation can participate directly in corpus retrieval through an index designed for its multi-vector structure.
The distinction affects latency accounting. A late interaction system used only after first-stage retrieval pays query encoding plus scoring over a bounded candidate set. An end-to-end late interaction retriever also pays candidate discovery and index traversal across the corpus.
Comparing only the final scoring operator hides these different workloads. Measurements need to include the complete retrieval path being replaced.
Evaluation needs ranking metrics and index metrics
A multi-vector model can improve relevance for a test set while imposing storage or latency costs that do not fit the serving target. Conversely, aggressive compression can make the index practical while changing enough token similarities to reduce ranking quality.
Useful evaluation therefore spans both sides of the design: retrieval metrics at the required cutoff, index size, query latency, memory use, and candidate-scoring cost. Corpus size, document length distribution, query length, hardware, and index settings need to remain visible in comparisons.
Single-vector retrieval remains attractive when compact storage and simple approximate search dominate the requirement. Cross-encoders remain attractive when a small candidate set permits full pairwise modeling. Late interaction fits between them by preserving token-level comparison without requiring document transformer encoding for every query.
That position is also its main constraint. Fine-grained matching survives because the system stores and searches more representation state. The extra vectors are not incidental overhead; they are the mechanism that keeps token-level evidence available after document encoding has been separated from the query.