A sliding key-value cache seems to offer a simple bound on transformer inference memory: retain the most recent tokens and evict the oldest entries as generation continues. That policy preserves local context, but it can change attention behavior more sharply than token age alone suggests. Some early positions can attract substantial attention even when their lexical content is not directly relevant to the current token. Removing those positions can disturb the distribution that later layers receive.
These persistent early positions are commonly described as attention sinks. They matter for cache design because a fixed memory budget does not require every retained token to come from one contiguous recent window. A small stable prefix can coexist with a moving window of recent tokens.
Attention mass does not follow token age
For one attention head, a query produces scores against cached keys and converts those scores into normalized weights. In simplified form:
attention(q, K, V) = softmax(q K^T / sqrt(d)) VThe softmax normalization couples every retained key. Removing a key does more than remove its associated value contribution: the remaining weights are normalized over a different set of positions.
A pure sliding cache assumes that old positions become expendable as they move far from the current token. That can be a useful locality assumption, but position age is not itself an attention score. If an early position repeatedly receives nontrivial mass, evicting it changes the normalization pattern presented to the value aggregation.
This effect is distinct from losing old semantic facts. A token can matter as an attention sink even when its text is not carrying a fact that the current output needs. Cache policies based only on semantic recency can therefore miss a structural role played by early positions.
A stable prefix changes the eviction boundary
Consider a cache budget of B token positions. A pure sliding policy keeps the newest B entries:
[t-B+1 ... t]A sink-preserving policy can reserve S entries for an initial prefix and use the remaining capacity for recent tokens:
[0 ... S-1] + [t-(B-S)+1 ... t]The total number of cached positions remains B. What changes is the eviction rule. The oldest entries inside the moving region are discarded, while the reserved prefix remains available to attention.
This is not equivalent to extending the cache. If S positions are reserved, the recent window becomes S positions shorter at the same capacity. The useful setting therefore depends on the model and workload: preserving more prefix positions leaves less room for recent context.
The reserved positions also need not be interpreted as a universal semantic prefix. Their value comes from preserving attention behavior observed for the model under the chosen inference setup. Treating an arbitrary set of early tokens as guaranteed sinks would overstate what the mechanism provides.
Position handling remains part of the cache contract
Evicting KV entries does not automatically define how positional information should behave. Transformer implementations can encode position through different mechanisms, and cache compaction must remain consistent with the model’s positional scheme.
A dangerous implementation shortcut is to move retained tensors into lower array indexes and assume that their logical positions have also changed. Storage position and model position are separate concepts. A cached key produced for a token at one logical position may already contain position-dependent transformation. Reindexing storage does not necessarily make that key equivalent to one originally computed at another position.
The safe invariant is narrower: cache management must preserve the positional semantics expected by the model and attention implementation. The exact bookkeeping is architecture-specific. A cache policy that works with one positional mechanism should not be generalized to another without checking how positions enter key and query computation.
Cache capacity and model context are different limits
A bounded KV cache controls retained attention state. It does not increase the context length supported by the model, nor does it guarantee behavior equivalent to full-context decoding.
Once entries are evicted, future tokens cannot attend to those cached keys and values unless the system reconstructs or represents their information through another mechanism. A sink-preserving window therefore trades exact full-history attention for bounded state while attempting to keep a subset of structurally useful early positions.
This distinction matters when evaluating long generation. A model can remain numerically stable under a bounded cache yet lose information that appeared outside the retained regions. Conversely, a failure caused by removing sink positions can occur even when the discarded text itself appears unimportant. Both effects can exist at the same time.
Evaluation needs a full-cache reference
A cache optimization is easier to assess when its output is compared with decoding that retains the full available history. The comparison should keep model weights, tokenization, prompts, decoding parameters, and numerical precision as controlled as practical.
For deterministic token selection, divergence can be inspected at the token or logit level. With sampling, exact generated text is a weak diagnostic because small probability changes can lead to different sampled continuations. Comparing next-token distributions or fixed-randomness runs gives a more direct view of the cache policy’s effect.
The useful comparisons separate several policies under the same capacity:
- a pure recent-token window;
- a stable prefix plus recent-token window;
- full-cache decoding where the reference fits available memory.
The result is not a universal ranking. Different models, prompts, heads, layers, and generation lengths can produce different attention patterns. The test establishes whether retaining a prefix reduces the specific degradation introduced by sliding eviction for the system being evaluated.
Sink preservation is a cache policy, not extra memory
Attention sinks expose a limitation in age-only eviction: old positions are not interchangeable merely because they are old. Reserving a small prefix changes which state survives under the same KV capacity, preserving selected early keys and values at the cost of a narrower recent window.
That makes the mechanism most useful when treated as an inference design choice with explicit boundaries. It does not recover arbitrary discarded context, extend the model’s supported positions, or make positional bookkeeping optional. It changes the composition of a bounded cache so that token age is not the sole criterion for retention.