Autoregressive LLM serving often repeats the same initial tokens across many requests. A fixed system prompt, tool schema, or document prefix can occupy thousands of tokens before request-specific text begins. Computing attention state for that identical prefix on every request repeats prefill work that has already produced the same cached keys and values under compatible execution conditions.
Prefix caching stores reusable attention state for such shared token prefixes. It changes the amount of prefill computation required for a cache hit, but it does not make arbitrary similar prompts interchangeable. The reusable unit is tied to exact model input state, not semantic resemblance.
Prefix reuse operates before divergent tokens
During causal transformer inference, each token can attend only to positions at or before its own position. Once a prefix has been processed, its key and value tensors do not depend on tokens appended later. That causal boundary permits a later request with the same prefix to begin from cached state and process only the uncached suffix.
Consider two token sequences:
request A: [P0 P1 P2 P3 A0 A1]
request B: [P0 P1 P2 P3 B0 B1 B2]If the first four tokens represent compatible cached state, request B can reuse that state through P3. Reuse stops at the first divergent token. The suffix beginning at B0 still requires normal model execution.
This property distinguishes prefix caching from response caching. A response cache returns an already computed result for a matching request. A prefix cache supplies intermediate model state and generation continues from that state.
Token identity matters more than text appearance
A cache lookup ultimately has to represent the token sequence and the model state that produced its stored tensors. Two prompts that look equivalent to a person can tokenize differently because of whitespace, Unicode representation, template formatting, or tokenizer configuration. Different token IDs imply a different prefix.
The reverse distinction also matters: comparing raw strings is not sufficient when a serving stack transforms messages into a model-specific template before tokenization. Cache identity should be derived at the representation boundary used by inference rather than from an earlier application object that may omit formatting details.
This makes deterministic prompt construction useful beyond readability. Stable ordering of tool definitions, stable serialization, and stable template output increase the chance that requests share a literal token prefix. Reordering equivalent metadata can destroy reuse even when the model receives the same information in a different order.
Cached state is model-specific
Key and value tensors are products of a particular model execution. Reusing them across incompatible weights is not valid merely because token IDs match. A model update can change every cached tensor associated with a prefix.
Cache identity therefore needs a model boundary. In systems that serve several checkpoints, adapters, or revisions, the key space must prevent state produced under one effective model from being consumed by another unless the serving implementation explicitly establishes compatibility.
Execution configuration can add further boundaries. Positional treatment, attention implementation, tensor layout, precision, and cache representation are implementation concerns that may affect whether stored state can be consumed directly. A cache format should be treated as an internal inference artifact unless the serving system documents a stable interchange contract.
Position is part of reusable attention state
A token’s cached key and value state is associated with its place in the sequence. Prefix reuse works naturally when the cached tokens occupy the same initial positions in a later request. It does not imply that a cached block can be moved to an arbitrary offset without adjustment.
This matters for applications that assemble prompts from reusable fragments. A tool schema cached at the start of one prompt is not automatically reusable when inserted after a different preamble. The fragment contains the same token IDs, but its positional context has changed.
Some inference engines implement block-oriented cache management and may support specialized mechanisms around positional state. Such behavior is engine-specific. The general prefix-caching contract remains narrower: reuse a matching sequence from the beginning until identity or compatibility breaks.
Cache granularity changes hit behavior
Serving engines commonly manage KV memory in blocks rather than as one allocation per complete prompt. Block granularity affects how much of a matching prefix can be reused and how much metadata the cache must track.
Suppose blocks hold four tokens and a request shares six leading tokens with a cached sequence. An implementation that reuses only complete blocks can consume the first four cached tokens and recompute the remaining two shared tokens with the suffix. A design that supports finer-grained reuse may capture more of the match but pays for more detailed bookkeeping.
The useful metric is therefore not only whether a request hit the cache. Reused token count, recomputed prefix length, and memory retained per reusable block describe the actual effect more precisely than a binary hit ratio.
Memory pressure can erase future reuse
Prefix caching trades retained KV state for reduced repeated computation. A cache entry that remains resident occupies memory that could otherwise hold active request state or other reusable prefixes. Under pressure, an engine may evict cached blocks and recompute them on a later request.
Eviction policy interacts with traffic shape. A very large prefix used rarely can consume substantial cache space while producing little reuse. A smaller prefix shared by many requests can be more valuable even if each individual hit saves fewer tokens.
This makes cache admission and eviction workload-dependent. Retaining every observed prefix is not inherently preferable. The serving system has to balance active inference memory, expected reuse, and the cost of reconstructing evicted state.
Shared prefixes can expose isolation boundaries
A cache that is shared across requests also becomes part of the serving system’s isolation model. Cache keys must prevent one request from receiving state associated with an incompatible model context. Multi-tenant systems may require stronger partitioning when prompt material or cache metadata can reveal information about another tenant’s workload.
Even when cached tensors are never returned directly, cache-hit timing can become observable. Whether that observation matters depends on the threat model, deployment boundary, and what information is encoded in shared prefixes. Treating prefix caching as a purely local optimization can miss these system-level constraints.
An implementation can choose to partition cache namespaces by tenant, model, security domain, or another boundary. Partitioning reduces cross-boundary reuse in exchange for clearer isolation. That choice belongs beside other inference security decisions rather than being hidden inside a generic caching layer.
Prefix caching changes prefill cost, not decode semantics
A successful cache hit avoids recomputing some prompt positions. It does not remove the autoregressive dependency of newly generated tokens. Each decode position still consumes prior attention state and produces new state for the next position.
The distinction is useful when interpreting latency. Requests with long shared prompts and short outputs can benefit substantially from reduced prefill work. Requests dominated by long generation may see a smaller end-to-end effect because decode remains sequential at the token dependency level.
Batching also interacts with this split. An inference engine can schedule cached and uncached requests together, but prefix reuse does not eliminate scheduling, memory bandwidth, or decode costs. It removes a specific class of repeated computation: reconstructing attention state for an identical compatible prefix.
Prefix caching is most predictable when prompt construction and cache identity are designed together. Stable token prefixes create reuse opportunities; explicit model and isolation boundaries keep those opportunities valid. The optimization is narrow, but that narrowness is useful: it saves prefill work without changing the token sequence or the generation rule applied after the cached prefix.