Prefix Caching

2 articles

Artificial Intelligence 24 Sep 2026 5 min read

Prefix KV Caching Reuses Only the Shared Token Prefix

A prefix cache hit ends at the first point where a new request can no longer reuse previously computed state. The reusable object is not a piece of source text in isolation. It is model state produced for an ordered token prefix under execution conditions that make that state compatible with the new request. For transformer inference, that state is commonly the key-value cache created during prefill. Reusing it can remove repeated computation for the shared prefix while leaving the divergent suffix to be processed normally.

Artificial Intelligence LLM Inference

Artificial Intelligence 24 Sep 2026 5 min read

Prefix Caching Reuses KV State for Identical Token Prefixes

Autoregressive serving often receives requests that start with the same long token sequence. A system prompt, tool schema, fixed document header, or other repeated context can cause the model to compute the same prefix attention state again for each request. Prefix caching targets that repeated prefill work by retaining compatible key-value state and attaching later requests to it. The reuse boundary is exact tokenized state, not semantic similarity. Two prompts that express the same idea but tokenize differently do not produce an interchangeable prefix cache entry.

Artificial Intelligence Transformers