Autoregressive serving often receives requests that start with the same long token sequence. A system prompt, tool schema, fixed document header, or other repeated context can cause the model to compute the same prefix attention state again for each request. Prefix caching targets that repeated prefill work by retaining compatible key-value state and attaching later requests to it.
The reuse boundary is exact tokenized state, not semantic similarity. Two prompts that express the same idea but tokenize differently do not produce an interchangeable prefix cache entry.
Reuse begins at a shared token boundary
During prefill, each transformer layer produces key and value tensors for the prompt tokens. For a request with token sequence:
P = [p0, p1, ..., pk]a later request that begins with the same sequence can reuse cached state associated with P and compute only the suffix that follows it:
request A = P + A_suffix
request B = P + B_suffixThe two requests may diverge immediately after P. Their subsequent KV state is request-specific, but the state for the common prefix can remain shared if the serving implementation supports that form of reuse.
This differs from response caching. Prefix caching does not return a stored model response. Generation still runs for the new request. The reused object is intermediate attention state produced by prior prefill computation.
Token identity is stricter than text identity
Cache compatibility is normally tied to the model input representation and execution configuration. Text that looks identical can fail to map to the same token sequence when tokenization inputs differ. Added special tokens, chat templates, whitespace, tool serialization, or tokenizer versions can change the resulting prefix.
A practical cache key therefore cannot safely rely on raw text alone. It must identify enough of the execution context to prevent state from one incompatible invocation from being treated as valid state for another. The exact key structure is implementation-specific, but the compatibility problem includes the model and the token sequence at minimum.
Model state also matters. KV tensors are outputs of a particular model parameterization. State computed by one checkpoint is not generally interchangeable with state from a different checkpoint merely because token IDs match.
Prefix blocks make partial matches usable
Serving engines often manage KV state in blocks or pages rather than one allocation per full prompt. A block-oriented cache can match a request against a sequence of cached prefix blocks and reuse the longest compatible run.
Conceptually:
cached blocks: [B0][B1][B2]
new request: [B0][B1][C2][C3]
reuse: [B0][B1]
compute: [C2][C3]The cache does not need the entire prompt to match. Reuse can stop at the first incompatible block, after which normal prefill continues. Block size affects the granularity of matching and storage management, while the attention semantics still depend on the ordered token prefix represented by those blocks.
A block hash can identify content, but a hash match is only an indexing mechanism. Correctness still requires the runtime to ensure that the referenced cached state belongs to a compatible model execution context. Hashing does not make otherwise incompatible KV tensors interchangeable.
Position handling is part of compatibility
Key and query representations may depend on token position through the model’s positional mechanism. Reusing a cached prefix assumes that the stored state corresponds to the positions assigned to that prefix in the new request.
For the common case in which both requests start with the same prefix at position zero, the positional alignment is direct. More general reuse schemes need explicit handling if they attempt to transplant state to different positions. A cache system cannot infer positional equivalence from matching token IDs alone when the model representation depends on position.
The same boundary applies to implementation-specific attention metadata. A serving engine may include sequence layout, cache format, quantization parameters, or other state needed to interpret stored KV tensors. Such metadata belongs to the cache contract even when it is not visible in the prompt.
Reuse reduces prefill computation, not decode dependence
Prefix caching primarily removes repeated work for tokens whose KV state has already been computed. It does not eliminate autoregressive dependence for newly generated tokens. After the reusable prefix and any uncached prompt suffix are processed, decoding still advances token by token under the model’s decoding procedure.
The effect therefore depends on workload shape. Repeated long prefixes expose more reusable prefill state than requests with unrelated prompts. A cache can also consume memory without producing useful hits when prefixes rarely repeat or entries expire before reuse.
Memory capacity, eviction policy, block granularity, and request locality determine how much reusable state remains resident. These are serving-system properties rather than changes to the transformer architecture.
Shared state requires isolation rules
A serving system must treat cached KV state as derived request data. Reuse across requests can create security and privacy concerns if cache keys, tenancy boundaries, or lifecycle rules allow one context to reference state that should not be shared.
The safe boundary depends on the deployment. A single-tenant service may permit reuse across all compatible requests. A multi-tenant service may partition caches by tenant, authorization domain, model instance, or another isolation key even when token prefixes match.
This partitioning can reduce hit rate, but hit rate is not the only correctness criterion. Cache reuse must preserve both model-state compatibility and the application’s data-isolation policy.
Prefix caching composes with other KV techniques
Prefix caching changes whether an existing prefix state must be recomputed and duplicated. Other mechanisms act on different dimensions. Grouped-query attention reduces the number of key-value heads produced by the model. KV quantization changes the representation width of stored state. Paged allocation changes placement and fragmentation behavior. Bounded attention can limit how much historical state remains addressable.
These mechanisms can coexist because they do not define the same boundary. A quantized cache can still reuse prefix blocks, and a model with fewer KV heads can still benefit from avoiding repeated prefill for a shared prefix. Compatibility checks must include any representation details that affect interpretation of the stored tensors.
Prefix caching is most useful when repetition exists before requests diverge. Its structural constraint is exact reusable model state: matching text is insufficient, matching meaning is irrelevant, and a cache hit is valid only when token sequence, positions, model state, representation, and isolation rules agree at the reuse boundary.