Autoregressive transformer inference normally keeps key and value states from earlier tokens so each new token can attend to prior context without recomputing those states. The cache grows with sequence length. For a long-running stream, that growth eventually becomes a memory constraint even when generation itself continues one token at a time.
KV cache eviction puts a bound on that state by discarding selected cached positions. The memory effect is straightforward: fewer retained key-value pairs occupy less cache space. The model effect is more subtle. Once a position is removed, later attention layers cannot use its cached key and value in the ordinary attention calculation. An eviction policy therefore changes both resource use and the effective attention history.
This makes cache retention a model-behavior decision, not just a memory-management detail.
Cache growth follows retained sequence state
During causal self-attention, a generated token produces a key and value for each relevant attention layer. At the next decoding position, the query for the new token is compared with cached keys, and the resulting attention weights combine cached values.
A simplified view is:
token t
-> compute query, key, value
-> query attends to retained keys from positions <= t
-> retain selected key-value states for token t + 1Without eviction, the number of cached positions rises as the sequence grows. Exact memory use depends on the architecture and serving representation: layer count, key-value head count, head dimension, numeric format, and any cache layout or quantization all matter. Grouped-query attention, for example, uses fewer key-value heads than ordinary multi-head attention with the same query-head count.
The key point is independent of those implementation details. If every prior position remains cached, per-sequence KV state increases with retained context length. A fixed cache budget requires some form of truncation, compression, offloading, recomputation, or selective retention.
A sliding window gives recency a fixed budget
The simplest bounded policy keeps only the most recent W positions. When a new position arrives after the window is full, the oldest cached position is evicted.
before: [41 42 43 44 45]
new: 46
after: [42 43 44 45 46]This creates a clear memory bound and matches workloads whose useful dependencies are mostly local. It also creates a precise limitation: information represented only through an evicted position is no longer directly available through that position’s key and value.
That limitation can surface far from the eviction point. A stream may contain an identifier, constraint, or earlier statement that becomes relevant thousands of tokens later. Keeping the latest tokens preserves local continuity but does not preserve arbitrary long-range access.
A model architecture that was designed or trained with local or sliding-window attention is a separate case. Its attention pattern already restricts which positions a query can access. Applying a serving-time window to a model that normally uses full causal attention changes the available context relative to its standard inference path.
Initial tokens can have unusual attention effects
Pure recency is not the only retention pattern used for streaming inference. Research on StreamingLLM reported an effect called attention sinks: retaining a small set of initial tokens together with recent tokens stabilized streaming generation for the evaluated autoregressive models, even after intermediate tokens were removed.
The relevant policy has two retained regions:
[initial tokens] ... evicted middle ... [recent window]This does not restore the missing middle context. It preserves selected initial positions that can receive substantial attention mass while also keeping local context near the current decoding position.
That distinction matters for implementation. A cache policy based on attention sinks should not be interpreted as a generic guarantee that the first few tokens contain the stream’s semantic facts. The observed role concerns attention behavior in particular models and setups. Application-level facts located in discarded positions can still become inaccessible.
A serving system should therefore treat sink retention as a model-dependent inference technique and validate it against the exact model family and workload rather than assuming that any fixed prefix is sufficient.
Attention scores can drive selective retention
Another family of policies tries to retain positions estimated to matter more than simple recency indicates. Attention-derived scores are one possible signal. A system can preserve positions that have received high attention, combine them with a recent window, or select cache entries at a finer granularity such as individual attention heads.
This changes the failure mode. A recency window deterministically loses old positions. A score-based policy can preserve old positions, but its selection criterion is based on evidence available at selection time. Future queries may need a position that looked unimportant earlier.
Consider a long interaction containing two old facts:
A: deployment region = ap-southeast-1
B: retry delay = 400 msIf recent attention repeatedly uses A but not B, a retention score may favor A. A later request about retry timing can still need B. Past attention is evidence about past use; it is not a guarantee of future relevance.
Head-level behavior adds another complication. Different attention heads can exhibit different access patterns. A cache method that applies one token ranking uniformly across all heads may discard state that is useful to a head specializing in a different pattern. Methods that distinguish retrieval-oriented heads from more local heads attempt to exploit this difference, but they also add model-specific analysis or calibration.
Position handling must match the model
Evicting cached entries does not mean the remaining tokens can be renumbered arbitrarily.
Transformer models encode position through mechanisms defined by the architecture, such as rotary position embeddings or other positional schemes. A serving implementation has to maintain position handling that is compatible with the model and with the cache transformation it applies. Treating retained entries as though they had always occupied a shorter contiguous sequence can alter attention calculations.
This is one reason KV eviction is not equivalent to deleting text from an input string and running ordinary inference on the shorter text. Cached states were computed under a particular positional context, and the serving method determines how subsequent positions interact with those retained states.
Framework support also matters. A cache policy implemented by a serving engine may rely on model-specific attention kernels, cache layouts, or position bookkeeping. The behavior of a research method cannot be inferred from its high-level retention rule alone.
Memory savings and semantic retention are separate measurements
A bounded cache can meet its memory target exactly and still damage output quality for a workload that depends on evicted context. Memory accounting therefore cannot serve as the only evaluation.
A useful evaluation separates at least three questions.
First, does the policy maintain the intended cache bound under realistic sequence lengths and batching conditions? This checks the systems objective.
Second, does it preserve the model behavior required by the application? Tests should include dependencies at different distances, especially facts or constraints that cross the eviction boundary. Aggregate text similarity can miss failures in exact identifiers, code symbols, numerical values, or instructions.
Third, what latency cost accompanies the policy? Selecting entries, moving cache blocks, computing retention scores, or using specialized kernels can add work. A method that stores fewer entries does not automatically reduce end-to-end latency in every serving stack.
These measurements should use the same decoding settings when comparing retention policies. Sampling differences can otherwise obscure whether an observed output change came from cache handling or token selection randomness.
Retention policy should follow the dependency pattern
Cache eviction is most defensible when the application has a known bound on useful history or when evaluation shows that selective retention preserves the dependencies that matter. Continuous feeds with strongly local structure can fit a recent-window policy. Workloads that require exact recall from arbitrary earlier positions place a much harder constraint on bounded caches.
External state can change that boundary. An application may extract durable facts into structured storage or retrieve relevant earlier material back into the prompt. That design moves some long-range dependency outside the raw KV cache. It does not make eviction lossless; it gives the system another path for restoring selected information.
The practical boundary is simple: a KV cache is model state for attention, not a durable memory store. Once a serving system evicts entries, it should assume those positions are unavailable to ordinary later attention unless another mechanism explicitly restores their information. A fixed memory budget is therefore also a fixed policy about which parts of the past remain computationally reachable.