A KV cache entry is not a reusable representation of arbitrary text that happens to look similar. For an autoregressive transformer, cached keys and values are intermediate states produced for a specific token prefix under a specific execution context. Reusing them is valid only when the new request reaches the same state boundary.
That boundary is stricter than matching visible characters. Tokenization, token order, position handling, model identity, adapter state, and other inputs that affect hidden states can all determine whether a cached prefix still represents the computation required by the new request.
The cache stores state after a token history
During causal self-attention, each layer produces key and value vectors for token positions processed so far. At decoding step t, attention can use cached states from earlier positions instead of recomputing them from the full prefix.
For one layer, the relevant structure can be written schematically as:
K_cache = [K_0, K_1, ..., K_(t-1)]
V_cache = [V_0, V_1, ..., V_(t-1)]A new token contributes another key and value pair. The cache therefore represents a computation path through an ordered token sequence. It is not equivalent to a semantic embedding of the prefix.
Prefix reuse extends this idea across requests. If two requests begin with the same eligible token sequence, a serving system can retain the corresponding layer states and begin new computation after the shared prefix. The skipped work is the transformer computation that produced those cached states, subject to the runtime’s cache design and supported execution path.
The cache does not imply that all request processing disappears. Tokenization, cache lookup, metadata handling, unmatched suffix processing, sampling, and later decoding still remain.
Text equality and token equality are separate checks
A serving layer commonly receives text, while the model consumes token IDs. Cache identity has to be established at the representation used by the model.
Two strings that appear close to a human reader can tokenize differently because of whitespace, Unicode representation, punctuation, template formatting, or tokenizer configuration. Once token IDs differ at a position, subsequent hidden states generally belong to a different token history.
The converse also matters operationally: a cache key based only on raw text can be too weak or too strict depending on where normalization and templating occur. If a chat template inserts control tokens before tokenization, the effective prefix is the rendered token sequence, not merely the user-visible message.
A sound cache design therefore anchors identity to the actual model input sequence or to a key that deterministically represents it. Hashing can make lookup compact, but the hash is only an indexing mechanism. Correctness still depends on the data included in the key and on collision handling appropriate to the implementation.
A mismatch cuts off reusable state at that position
Suppose two requests produce these token sequences:
A: [31, 44, 90, 12, 77, 18]
B: [31, 44, 90, 63, 52]Their common prefix has three tokens. States corresponding to those first three positions can be candidates for reuse. States produced after token 12 in request A cannot stand in for states after token 63 in request B.
Causal attention creates this boundary because the representation at a position depends on the preceding allowed context. A mismatch changes the history presented to later positions, so a cache cannot simply resume after a divergent region and treat a later textual match as continuous state.
Block-oriented caches add granularity to this boundary. A runtime may index fixed-size token blocks rather than individual positions. In that case, a partial matching block may require fresh computation even when some tokens inside it match. This is an implementation property of the cache layout, not a change in transformer semantics.
Position state is part of compatibility
Matching token IDs are necessary in many prefix-cache designs, but token IDs alone do not describe every model input that can affect cached states.
Position-dependent computation is a central example. Transformers can encode position through mechanisms such as rotary position transforms, absolute position embeddings, or other model-specific rules. If the same token sequence is interpreted at different positions under a model path where position changes key or value states, cached tensors from one position assignment cannot be assumed valid for the other.
Serving systems can avoid this ambiguity by defining cache entries at a known prefix position and carrying the required position metadata with the entry. The exact compatibility rule depends on the architecture and runtime. A cache implementation should follow the model’s actual position semantics rather than treating position as generic metadata that is safe to omit.
The same caution applies to attention structure. If an execution path changes which prior positions a token may attend to, identical token IDs can still produce different states. Cache reuse must preserve the attention semantics under which the cached tensors were generated.
Model and adapter state belong in the cache boundary
A KV cache is tied to the parameters that produced it. Reusing states across different model weights is not a valid shortcut merely because tokenizer vocabulary and token IDs match.
This becomes relevant when a serving process hosts multiple model revisions or parameter-efficient adapters. If an adapter changes projections or other computations that contribute to hidden, key, or value states, cache entries generated under one adapter state cannot automatically be used under another.
The cache namespace or lookup key should therefore separate execution states that can produce different cached tensors. Depending on the system, that can include model revision, adapter identity, relevant model configuration, and any request-level feature that changes the computation before or within the cached prefix.
Not every serving option belongs in that identity. A sampling temperature applied after logits are produced, for example, does not retroactively change keys and values already computed for the prefix. Cache compatibility should be based on causal inputs to cached state, not on every request field indiscriminately.
Cache hits can be partial without being approximate
Prefix caching is sometimes described as an all-or-nothing hit, but a trie, radix structure, or block index can expose the longest cached prefix that matches a request. The request can reuse that exact portion and compute the remaining suffix normally.
This is exact reuse when the cached tensors are the same states the active model path would have produced for that prefix. No semantic similarity threshold is required. Approximate retrieval belongs to a different mechanism: retrieving related documents or vectors can change model context, while prefix-cache reuse avoids recomputation of an already identical computation prefix.
Keeping those mechanisms separate also clarifies metrics. A request-level hit rate can hide whether only a small prefix was reused. Token or block reuse counts describe saved prefix coverage more directly, although they still do not map one-to-one to latency because kernel scheduling, memory movement, cache lookup, and batch composition also contribute to serving time.
Eviction changes availability, not validity
A valid prefix may be absent because its cache entry was evicted. Capacity limits, recency policies, block pressure, or explicit invalidation can remove reusable state without changing the correctness rule.
This distinction matters when diagnosing misses. A compatibility miss means the available entry does not represent the required computation state. A capacity miss means a compatible entry could have been used but is no longer resident. Combining both into one cache-miss counter obscures different causes.
Memory accounting also needs to reflect the cache’s physical representation. KV state is stored across transformer layers and token positions, with size affected by model architecture, cache dtype, head structure, and runtime layout. A high reuse opportunity does not guarantee that retaining every prefix is a sensible allocation policy.
The durable boundary is state identity: prefix reuse is safe when the cached tensors correspond to the same computation the active request requires at that prefix. Textual resemblance, repeated phrases, or matching suffixes do not establish that identity. A serving system gets reliable reuse by making token history and every state-defining execution input explicit in its cache key and namespace.