A prefix cache can remove repeated prefill work without changing the model output, but only when the cached key and value states represent the same prefix context that the new request would have produced. Matching visible text is not enough. The serving path ultimately operates on token IDs, positions, model parameters, and implementation-specific attention state.
This boundary makes prefix caching different from a generic text cache. A text cache stores a result associated with an input. A KV prefix cache stores intermediate states whose validity depends on the computation that created them.
The cache represents states, not source text
For a transformer layer, the cached entries are derived from hidden states after the prefix has already passed through earlier computation. In a simplified attention view, a token state h_i produces a key and value:
k_i = W_K h_i
v_i = W_V h_iThe actual computation can include multi-head or grouped-query layouts, positional transforms, normalization, and other architecture-specific details. The key point is that k_i and v_i are not determined by the visible character string alone.
Two requests that render the same text can tokenize differently if tokenizer configuration or preprocessing differs. A request that contains the same token IDs at different positions can also produce incompatible attention state in architectures where position affects the cached representation. Reusing entries across either mismatch changes the state presented to later tokens.
A safe cache key therefore has to identify the effective model input and every serving condition that can alter the cached representation. The exact fields are implementation-specific; the invariant is not.
Prefix identity extends through token order
Autoregressive attention makes the state for a prefix position dependent on the context available at that position. If two requests share token IDs only after an earlier mismatch, the later matching span does not generally provide an independent reusable cache segment.
Consider these token sequences:
A: [11, 24, 31, 42, 57]
B: [11, 24, 99, 42, 57]The first two positions form a common prefix. Tokens 42 and 57 match later, but their hidden states can differ because the preceding context differs. A prefix cache can reuse the initial common segment; it cannot infer that later equal token IDs imply equal KV states.
This is also the reason block-oriented caches track ordered prefix blocks rather than treating blocks as an unordered collection. A block match is useful only when all prior state required by that block is compatible.
Position handling is part of validity
Position information may enter attention in several forms. With rotary position embeddings, for example, position-dependent rotations are applied to query and key components. Other architectures can use different positional mechanisms.
If a cached prefix is moved to a different logical position, direct reuse may no longer represent the computation that an uncached prefill would perform. Some serving implementations can support specialized transformations or cache layouts, but such behavior is an implementation property, not a general guarantee of KV caching.
The conservative boundary is simple: cached entries should retain the positional interpretation under which they were produced unless the serving implementation explicitly defines a correct conversion.
Attention masks and sequence structure can create a similar constraint. A cached state produced under one visibility pattern is not automatically valid under another visibility pattern, even when the token IDs match.
Model identity belongs in the cache boundary
KV states are products of model parameters. Loading a different checkpoint, adapter, or parameter set can change the projected keys and values for the same token sequence.
That means a serving system must prevent entries created by one effective model configuration from being consumed by an incompatible configuration. Depending on the stack, effective identity can include the base checkpoint plus adapters or other components that alter forward computation.
Precision and kernel details require a more careful distinction. Different kernels or numeric formats can produce numerically different states without necessarily defining a different semantic model. Whether cached buffers can be shared across those execution paths depends on representation compatibility, memory layout, and the serving engine’s guarantees. It should not be inferred from model name alone.
Prefix reuse and decode caching solve different repetition
Ordinary autoregressive decoding retains KV states from tokens already processed in the current request. At the next generation step, the model computes state for the new token instead of recomputing the entire preceding sequence.
Prefix caching extends reuse across requests. A system recognizes that a new request begins with a prefix for which compatible states already exist and skips some corresponding prefill computation.
The distinction affects lifecycle and isolation. A per-request decode cache can be discarded when generation ends. A cross-request prefix cache needs an eviction policy, cache identity rules, and isolation controls appropriate to the data it retains. Those operational choices do not change the mathematical condition for reuse, but they determine whether a mathematically compatible entry is also acceptable to share.
Longer matches do not make invalid entries valid
A high cache-hit rate can look attractive while hiding weak identity rules. If the cache considers two incompatible contexts equivalent, a larger hit rate means more requests consume states that should have been recomputed.
Validation should therefore compare cached execution with the uncached path under the same model and request conditions. The relevant question is whether reuse preserves the serving engine’s expected output behavior within its documented numeric constraints, not merely whether a lookup succeeds.
It is also useful to distinguish cache-key collisions from deliberate equivalence. A hash can compactly identify a token block, but equal hash values are not a mathematical proof that the underlying contexts are equal unless the implementation verifies enough information or accepts the collision risk by design. Hash construction and collision handling belong to the cache implementation, not to transformer semantics.
Prefix KV caching is safe at the boundary where an entry still denotes the intermediate state that the current request would have produced. Token equality is necessary for ordinary exact-prefix reuse, but serving systems also have to preserve position, preceding context, effective model computation, and any execution constraints encoded by the cache representation. Past that boundary, reuse stops being an optimization of the same computation and becomes a different computation.