A prefix cache hit ends at the first point where a new request can no longer reuse previously computed state. The reusable object is not a piece of source text in isolation. It is model state produced for an ordered token prefix under execution conditions that make that state compatible with the new request.
For transformer inference, that state is commonly the key-value cache created during prefill. Reusing it can remove repeated computation for the shared prefix while leaving the divergent suffix to be processed normally.
Reuse follows token identity and prefix order
Consider two token sequences:
A = [t0, t1, t2, t3, a0, a1]
B = [t0, t1, t2, t3, b0, b1]The first four token positions form a common prefix. If their cached state is compatible, request B can begin from the KV state after t3 instead of recomputing those four positions.
The useful boundary is positional. A matching token span later in the request does not provide the same property. Autoregressive attention state for a token depends on the preceding context, so identical token IDs reached through different prefixes do not generally identify interchangeable KV state.
This also means text equality is not the strongest cache key. Tokenization, model identity, adapter state, multimodal inputs, and other execution-specific inputs can affect the state associated with a nominally similar prompt.
Block granularity can shorten a logical match
Inference engines often manage KV memory in blocks. A request may share more tokens than the cache can address as a reusable unit.
With a four-token cache block, suppose an existing request contains:
[A B C D] [E F G H]and a new request begins:
[A B C D] [E F X Y]The logical common prefix has six tokens, but a cache that registers only complete four-token blocks can reuse the first block directly. The second block is not identical as a complete cache unit.
This boundary is an implementation property, not a transformer rule. An engine with finer prefix-match granularity can expose a different reusable boundary while preserving the same model semantics.
vLLM’s current prefix-caching design hashes cache units from their token IDs, the parent-prefix hash, and additional identity fields. Its documentation also distinguishes prefix-match granularity from physical KV block size in configurations that support finer matching.
A hit removes prefill work, not decode work
For a prompt of P tokens with a reusable prefix of C tokens, the uncached prompt portion contains:
P - Ctokens. Prefix caching can avoid model computation for the cached portion during prefill. It does not remove the autoregressive forward passes required for newly generated tokens.
This distinction changes the workload profile in which the optimization matters. Repeated long prefixes create substantial reusable prefill. A request dominated by a long generated continuation can still spend most execution time in decode even after a large prefix hit.
The cache also does not alter sampling semantics by itself. When the reused KV state is exactly the state that recomputation would have produced under compatible execution conditions, reuse substitutes stored intermediate state for repeated prefill computation.
Cache identity extends beyond visible tokens
A safe cache key has to separate states that are not interchangeable. A practical engine may therefore include more than token IDs in cache identity.
Examples include an adapter identifier, a hash representing multimodal input, cache-isolation data, or other model-execution parameters. The exact fields are implementation-specific. Their purpose is the same: prevent a syntactically matching token prefix from selecting state produced under incompatible conditions.
A chained hash is one way to preserve prefix identity. Conceptually, a block key can be represented as:
H_i = H(H_{i-1}, tokens_i, extra_i)where H_{i-1} commits to the preceding prefix, tokens_i identifies the current cache unit, and extra_i carries additional execution identity.
Including the parent value matters. Two blocks containing the same local token IDs but reached through different preceding tokens should not automatically map to the same autoregressive state.
Eviction turns reuse into a temporal property
A previously computed prefix is reusable only while its cached state remains available. Finite KV memory therefore makes prefix reuse dependent on cache residency as well as request similarity.
An engine can retain inactive blocks for later reuse and evict them when memory is needed. A later request may have an exact prefix match at the token level yet receive no cache hit because the corresponding blocks have already been reclaimed.
Reference tracking adds another boundary. State currently used by an active request cannot be treated exactly like an unreferenced cached block that is eligible for eviction. Cache managers commonly track both identity and liveness to decide which physical blocks can be reassigned.
As a result, hit rate is a property of workload ordering, memory pressure, eviction policy, and matching granularity. It is not determined by prompt duplication alone.
Shared caches create an isolation boundary
Cross-request reuse can expose information through timing. A request that hits a large cached prefix may reach its first generated token sooner than an otherwise equivalent miss. In a shared service, latency can therefore reveal whether particular state was already resident unless the cache is partitioned or otherwise isolated.
vLLM documents cache salting for this boundary. A salt contributes to cache identity so requests with different salts do not share cached prefix blocks. This trades some cross-request reuse for isolation between trust domains.
Hash selection is another implementation concern. A cache lookup that relies on hashes must account for collision risk because selecting the wrong KV state is not equivalent to an ordinary metadata-cache collision. Current vLLM documentation uses SHA-256 as its default prefix-caching hash and warns that non-cryptographic alternatives change the collision-risk profile.
Prefix reuse stops where state equivalence stops
Prefix caching is exact state reuse, not approximate prompt similarity. Its useful span is bounded by the longest compatible token prefix that the engine can address and that still resides in cache.
That boundary can be shorter than the visible shared text because tokenization or cache units differ. It can disappear because blocks were evicted. It can also be intentionally narrowed by isolation metadata.
The optimization is therefore best described in terms of state equivalence: cached KV entries replace repeated prefill only for positions whose prior context and execution identity match the state already stored. Everything after that boundary returns to ordinary model execution.