A bounded KV cache seems to invite a simple eviction rule: once the cache is full, discard the oldest key-value pair and keep the most recent tokens. For some decoder-only transformers, that rule can degrade generation sharply after the sequence moves beyond the retained window. The failure is not explained only by missing old semantic content. Early tokens can receive substantial attention even when their text carries little useful information for the current prediction.
This behavior is commonly called an attention sink. It changes the design of streaming inference because a cache policy that preserves a small initial prefix plus a recent sliding window can behave very differently from a pure sliding window with the same general memory constraint.
Softmax attention can make low-value positions structurally useful
For a query vector q, attention over cached keys assigns normalized weights:
a_i = exp(q · k_i) / sum_j exp(q · k_j)Every query distributes its attention mass across the positions visible to that head. A token does not need to carry useful semantic content to receive a large weight. Its key can become a convenient destination for attention mass that the head does not direct toward more informative positions.
The first few positions are unusual because they are visible to nearly every later token under causal attention. During model optimization, those positions can acquire key representations that repeatedly attract attention. The resulting sink behavior is a property of the trained attention pattern, not a guarantee attached to a specific token string.
This distinction matters for cache eviction. Removing an old token normally means removing information that can no longer be attended to. Removing a sink can also alter the normalization pattern of attention heads that had consistently allocated weight to it. The remaining tokens then receive a different distribution of attention mass even if all semantically relevant recent text is still present.
A pure sliding window changes the attention domain
Suppose a decoder keeps at most W cached positions. Under a pure sliding-window policy, token t can attend only to the most recent cached range:
[t - W, ..., t - 1]Once the sequence exceeds W, every decoding operation evicts another old position. If the model relies on early sink positions, those keys and values disappear permanently.
A sink-aware cache instead reserves S slots for initial tokens and uses the remaining slots for recent context:
[prefix of S tokens] + [most recent W - S tokens]The memory footprint remains bounded. The semantic history available in the recent window becomes slightly shorter because the prefix consumes cache slots, but the early attention structure remains available. This is a concrete memory-allocation trade: S positions are spent on persistent prefix state rather than additional recent state.
The prefix size is model-dependent. Retaining one initial token is not a universal contract, and retaining an arbitrary fixed count does not guarantee stable output. The relevant question is whether the attention heads in the target model depend on persistent early positions under the intended decoding regime.
Cache eviction and position handling are separate concerns
Keeping sink tokens does not make positional bookkeeping optional. A KV cache stores representations produced with positional information already incorporated according to the model architecture. Evicting middle positions from the cache does not imply that surviving entries can be relabeled freely.
With rotary position embeddings, for example, query and key rotations depend on position. An implementation that changes position indices during streaming can therefore alter query-key relationships independently of the cache policy. Some serving systems use position transformations or cache-specific handling to support bounded attention, but those rules are implementation-specific and must match the model and inference method.
This creates two independent failure sources that can look similar in generated text:
- the cache removed keys or values that the model still depends on;
- positional treatment no longer matches the assumptions used for the retained states and new queries.
A cache experiment is easier to interpret when only one of these dimensions changes at a time. Comparing full-cache decoding, pure window eviction, and prefix-plus-window eviction with identical position handling isolates the effect of retained sink states more cleanly.
Attention sinks do not preserve arbitrary long-range facts
A persistent prefix should not be confused with general long-context memory. Sink tokens can stabilize attention behavior without storing the semantic details of every evicted token.
If a fact appears far behind the recent window and its KV states have been discarded, a sink-aware bounded cache does not reconstruct that fact. The method is suited to streaming generation where bounded memory is required and very old content is allowed to become inaccessible. Tasks that require retrieval from arbitrary earlier positions need another mechanism, such as a larger cache, selective retention based on content, external retrieval, or a model architecture designed for that access pattern.
This boundary also affects evaluation. Perplexity or next-token quality on a streaming sequence tests local predictive stability, while a long-range recall probe tests access to distant information. A cache policy can perform well on the first and fail the second without contradiction.
Sink behavior can differ across heads
Multi-head attention does not require every head to use the prefix in the same manner. Some heads may place substantial mass on initial positions, while others concentrate on recent tokens or content-dependent locations. Averaging attention weights across all heads can hide this structure.
For cache analysis, per-head attention patterns are therefore more informative than a single layer-wide mean. A small number of heads with strong sink behavior can matter even when the aggregate fraction of attention assigned to the prefix appears modest.
The same caution applies across layers. Early layers and later layers can use cached positions differently. A serving optimization that applies one eviction assumption to every layer and head is making a stronger claim than an optimization based on observed attention structure for the target model.
Bounded memory changes the evaluation horizon
A cache policy may look correct on prompts shorter than its capacity because no eviction has occurred. The meaningful boundary begins when generation passes the point at which old entries must be removed.
Tests should therefore include sequences substantially longer than the configured cache and compare behavior before and after eviction starts. Useful signals include next-token loss, output stability under controlled prompts, attention allocated to retained prefix positions, and the exact number of KV entries held over time.
Memory accounting should include the persistent prefix. If the total limit is W and S slots are reserved, the recent portion has capacity W - S; treating the recent window as W as well silently increases the cache budget.
A further comparison against full-cache decoding provides a reference rather than an expectation of exact equivalence. Bounded eviction intentionally removes context, so divergence can be legitimate once discarded content matters. The narrower engineering target is to avoid extra degradation caused by removing structurally significant sink positions.
Attention sinks expose a subtle property of transformer inference: cache entries can matter for the geometry of attention even when their token content appears irrelevant to the current text. A bounded cache policy should therefore be evaluated as a change to the model’s attention domain, not merely as storage reclamation.