A decoder that keeps only the newest key-value states can degrade even when its cache still contains enough recent text for the immediate task. In some transformer models, early token positions attract substantial attention across later decoding steps. Evicting those states changes the attention distribution, not just the amount of accessible history.
Attention sink retention addresses that specific failure mode. A bounded cache preserves a small prefix of initial key-value states together with a moving window of recent states. Tokens between those regions can be discarded, keeping cache size bounded as generation continues.
A sliding window changes the softmax candidate set
For one attention head, a query at position t assigns weights to cached keys through a softmax over attention scores:
a_i = exp(s_i) / sum_j exp(s_j)The denominator is taken over the keys that remain available to that query. Cache eviction therefore changes the normalization set. Removing an old key does more than remove its value contribution; it also removes its score from the denominator and redistributes probability mass across the retained positions.
A pure recent-token window repeatedly removes the oldest surviving state. If early positions act as attention sinks, those states disappear once the sequence moves beyond the window boundary. Subsequent attention is then normalized over a set that differs systematically from the one the model encountered before eviction.
This effect is distinct from ordinary loss of distant semantic content. An early sink token can matter to attention allocation even when its text carries little task-specific information.
Sink tokens are not a second long-term memory
An attention sink is a position that receives disproportionate attention from later queries. The term describes an attention pattern, not a guarantee that the corresponding value vector contains useful facts from the distant context.
That distinction matters for cache design. Retaining sink states can preserve an attention pattern while still discarding most old content. It does not give the model access to facts encoded only in evicted tokens.
A bounded cache with sink retention can be represented as two regions:
[prefix sink states] ... evicted states ... [recent states]If the prefix contains S states and the recent window contains W states, the retained cache has at most S + W positions per layer after the window fills. Memory use remains bounded with respect to generated sequence length.
The exact storage cost still depends on model architecture, data type, layer count, batch size, and whether key-value heads are shared. Sink retention changes which positions occupy the cache; it does not change the shape of each stored key or value tensor.
The retained prefix must keep its positional meaning
Cached key and value states are products of the model’s positional mechanism as well as token content. A streaming implementation cannot generally move cached states to arbitrary logical positions and assume equivalent attention behavior.
With rotary position embeddings, for example, positional rotation has already affected cached keys. Some streaming schemes apply position transformations or use model-specific cache handling to make a bounded window behave consistently. Such transformations are implementation details that must match the model’s positional formulation.
A safer conceptual rule is that eviction policy and position policy are separate concerns. Keeping the first few cache entries does not by itself specify how positions for the moving window are represented.
This boundary also makes cache compatibility model-specific. A policy that works for one architecture is not automatically valid for another model with a different positional encoding, attention mask, or cache layout.
Sink count is a model property to measure
There is no universal prefix length that identifies every useful sink state. The first token is a common location for sink behavior in decoder models, but attention can also concentrate on several initial positions. Tokenization and special-token placement affect which positions occupy that prefix.
Allocating too few sink slots can remove states that participate in the pattern the policy is meant to preserve. Allocating more slots consumes cache capacity that could otherwise hold recent context. With a fixed total budget, increasing S usually reduces the available recent window W.
The useful split should therefore be evaluated for the exact model revision and serving format. A chat template can insert system markers or other special tokens before user text, so a fixed count refers to token positions after templating and tokenization, not to a fixed number of visible words.
Per-head behavior can differ as well. Some heads may depend strongly on early positions while others focus on local or content-based locations. A uniform cache policy is simpler to implement, but it can retain states that some heads do not need.
Bounded cache quality has two separate limits
Sink retention can address instability caused by removing high-attention prefix states. It cannot recover arbitrary dependencies from discarded middle positions.
Consider a generation request that must copy an identifier introduced far before the recent window. If the identifier’s state has been evicted and is not represented elsewhere in retained context, preserving sink tokens does not restore that information. The cache remains intentionally lossy.
This creates two evaluation dimensions. One checks whether generation remains stable as the stream extends far beyond the cache capacity. The other checks whether tasks requiring distant content still succeed. A system can perform well on the first dimension and fail on the second.
Comparisons also need a full-cache reference under the same prompt and decoding configuration. Without that reference, a change caused by cache eviction can be confused with ordinary sampling variation or prompt sensitivity.
Eviction must respect shared cache structures
Serving runtimes often store key-value states in blocks rather than one allocation per token. A logical policy such as “keep the prefix and newest window” may therefore map to block retention, partial-block handling, or copying between cache pages.
The arithmetic cache bound does not capture those allocator details. If a block contains both retained and evicted positions, the runtime may keep extra states until the block can be reclaimed, or it may compact data into another block. Either choice can affect memory traffic and fragmentation.
Batching adds another constraint. Requests at different sequence lengths have different moving-window boundaries, so their logical eviction points do not advance together. A cache manager must track retention per sequence rather than infer one global boundary from the batch.
These mechanics do not alter the attention-sink concept, but they determine whether a bounded policy produces the expected memory behavior in an actual inference engine.
Treat sink retention as an attention-preservation policy
Attention sink retention is most precise when described as selective state preservation. It keeps a small set of early key-value states because removing them can disturb later attention, while a recent window supplies local context for ongoing generation.
That mechanism has a narrow boundary. It can make fixed-size streaming caches behave closer to full-cache inference for models that exhibit the relevant sink pattern, but it does not turn a bounded cache into full long-context memory. Developers still need separate handling for tasks that require retrieval from content outside the retained regions.