Autoregressive transformer serving often repeats the same prompt prefix across requests: a system message, a long document header, or a fixed tool schema may precede user-specific text. Prefix caching stores the key-value state produced by that shared prefix so a later request can resume computation from the cached boundary instead of recomputing the entire prefix.
The useful boundary is narrower than “similar prompts.” Reuse depends on the exact token sequence and on model state that affects the cached activations. A one-character text edit may preserve most tokens, shift tokenization near the edit, or change every token after a formatting boundary. The cache can only reuse the portion whose effective input is still identical.
The cached object is intermediate model state
During causal self-attention, each processed token contributes keys and values that later tokens can attend to. For a transformer layer, a simplified representation is:
K_prefix = [k_1, k_2, ..., k_n]
V_prefix = [v_1, v_2, ..., v_n]After tokens 1..n have been processed, generation of token n+1 can use those tensors without recomputing keys and values for the prefix. A serving system can retain that state after one request and attach it to another request that begins with the same cache-compatible prefix.
This differs from caching final text. The stored state is tied to a particular model execution context. It represents activations after tokenization, embedding, positional treatment, and all transformer layers up to the cache boundary.
The practical gain applies to prefill computation for reused tokens. Decode still has to process newly generated tokens, and any uncached suffix still requires prefill.
Text equality is not the real cache key
Two strings that appear equivalent to a person can tokenize differently. Whitespace, Unicode normalization, template formatting, special tokens, or chat-role markers can alter the token IDs supplied to the model. Prefix reuse therefore has to be reasoned about at the token and execution-state level, not from rendered text alone.
Consider two requests:
A = [system tokens][document tokens][question A tokens]
B = [system tokens][document tokens][question B tokens]If the first two segments produce the same token IDs under the same execution conditions, their cached state can form a shared prefix. The request-specific question begins a new suffix.
Moving a request identifier, timestamp, or other volatile field near the front shortens that common prefix:
A = [system][request-id A][document][question A]
B = [system][request-id B][document][question B]Even when the document is identical, the divergence appears before it. A simple prefix cache cannot jump over the differing region and resume reuse at a later identical region, because later hidden states depend on preceding tokens.
This makes prompt layout a serving concern. Stable material placed before volatile material exposes a longer contiguous prefix for reuse without changing the model architecture.
Positional treatment is part of compatibility
Cached keys and values were produced for tokens at specific sequence positions under the model’s positional mechanism. Reusing them at different effective positions is not generally equivalent to recomputing the sequence.
For models using rotary position embeddings, position-dependent rotation is incorporated into attention representations. Other positional schemes impose their own state or indexing constraints. A serving engine may support transformations or cache-management strategies for a particular architecture, but that behavior is implementation-specific.
The safe invariant is that a cache entry is valid only under the positional semantics assumed when it was created. Concatenating arbitrary cached segments is not justified merely because their token IDs match locally.
This also constrains truncation. Removing tokens from the beginning of a context can change the effective position or attention context of retained tokens. A cache created for the original sequence cannot automatically be treated as state for the truncated sequence.
Model identity extends beyond weight filenames
A KV cache encodes activations produced by model parameters. Changing those parameters changes the functions that generated the stored keys and values. Cache entries must therefore be isolated across model revisions unless the serving implementation has a specific compatibility contract.
The same principle applies to execution settings that alter hidden states or attention behavior. Adapter selection, architecture configuration, attention masks, multimodal embeddings, or other cache-relevant inputs can make an otherwise identical token prefix incompatible.
Not every request option belongs in the cache identity. A sampling temperature applied only after logits are produced does not retroactively change keys and values already computed for the prompt. By contrast, an adapter that changes transformer projections can change cached tensors directly.
A useful cache key consequently represents the inputs and model state that determine cached activations, rather than every field in the request object.
Prefix length and block boundaries change realized reuse
Serving engines often manage KV memory in blocks or pages rather than as one allocation per request. A prefix cache built on those units may only reuse complete cache blocks, leaving a short unmatched tail to be recomputed. Exact behavior depends on the serving engine.
This creates a distinction between logical common-prefix length and realized cache reuse. Two requests may share 2,050 tokens, while an implementation with fixed cache blocks can expose a slightly smaller reusable region if only complete blocks are shareable.
Block-level hashing also makes lookup practical: the identity of a later block can incorporate the identity of preceding blocks plus the tokens or other relevant data in the current block. That preserves the causal prefix relation. A block with identical local tokens is not considered interchangeable when its preceding context differs.
Cache metrics should therefore report actual reused tokens or blocks rather than infer reuse from prompt similarity.
Reuse saves prefill work but retains memory state
Prefix caching exchanges recomputation for retained KV state. A popular prefix may reduce repeated prefill computation while occupying accelerator or host memory for as long as the cache keeps it resident.
The memory cost grows with cached sequence length, layer count, KV-head configuration, head dimension, data type, and the number of distinct prefixes retained. Exact byte formulas depend on architecture and storage format, especially when an engine quantizes or offloads cache state.
Eviction policy then becomes part of serving behavior. A cache with insufficient capacity may repeatedly evict a large prefix before its next use, producing little reuse despite a high theoretical common-prefix rate. Conversely, retaining many low-frequency prefixes can displace a small set of frequently reused entries.
This is a workload property rather than a model-quality property. The model can produce identical outputs with or without prefix caching when the cache is implemented as exact reuse of valid intermediate state; the difference is in serving work and resource occupancy.
Cache isolation can carry security significance
A shared serving system may process prompts from different users or tenants. Reusing KV state does not require exposing the original prompt text to another caller, but cache lookup, timing, accounting, and eviction behavior can still cross isolation boundaries if the implementation shares entries globally.
The appropriate boundary depends on the threat model. Some systems may partition cache entries by tenant or authorization context. Others may permit sharing only for provider-controlled prefixes such as a fixed system prompt. Content-derived cache keys alone do not establish authorization to share state.
Cache lifecycle also matters during model updates and policy changes. Stale entries should not survive into an execution context that no longer satisfies the identity conditions used when they were created.
Prefix caching is most predictable when treated as memoization of a causal execution prefix, not as semantic matching. The reusable unit is the longest contiguous prefix whose tokens, positions, model state, and other cache-relevant inputs still describe the same computation. Everything after the first incompatible boundary belongs to a new execution path.