A fixed-size KV cache seems to invite a simple policy: keep the newest entries and evict the oldest. For some decoder transformers, that policy removes positions that later queries continue to assign substantial attention mass. Those early positions act as attention sinks, and dropping them can change the attention distribution far more than their semantic content suggests.
The effect matters specifically at inference time when a serving system truncates cached keys and values. It is not a general claim that the first token is semantically privileged, nor that every transformer exhibits the same pattern.
Softmax attention must place its mass somewhere
For one attention head, a query produces scores against cached keys and normalizes them:
[ a_i = \frac{\exp(s_i)}{\sum_j \exp(s_j)}. ]
The normalized weights sum to one over the positions visible to that query. In models that develop sink behavior, one or more early positions can receive persistently large weights even when those positions are not carrying content that appears relevant to the current token.
This creates an implementation boundary. Removing a sink key does more than shorten history. It changes the set over which softmax is normalized, so probability mass is redistributed among the surviving positions. The corresponding value contribution also disappears. Both changes occur before the next residual update propagates through later layers.
Attention-sink behavior was reported in the StreamingLLM work for several autoregressive model families. The result is empirical and architecture-dependent; it should not be promoted into a universal property of softmax transformers.
A recent-only window changes more than context length
Suppose a cache of capacity (W) retains positions
[ t-W+1, \ldots, t ]
for the current step (t). Once an initial sink position falls outside that interval, it is unavailable to every later query. A pure sliding window therefore changes both the accessible semantic history and the attention locations the model had come to use as sinks.
A sink-aware cache uses a different partition. It can retain a small fixed prefix while allocating the remaining capacity to recent positions. If (S) prefix entries are retained, the cache contains the prefix plus roughly (W-S) recent entries rather than (W) recent entries alone.
That policy does not restore arbitrary old context. Tokens evicted from the middle are still inaccessible. The retained prefix serves a narrower purpose: preserve positions whose removal can disturb attention behavior while bounding KV memory.
Sink tokens are not a substitute for retrieval
A retained sink position and a retained factual token solve different problems. A sink can absorb attention without encoding the old fact, instruction, or entity that a later query needs. Once a content-bearing KV entry has been evicted, keeping the sink does not reconstruct it.
This distinction prevents a common serving mistake. Stable streaming generation under a bounded cache is not equivalent to full-context generation. A model can maintain coherent local behavior while losing access to details outside the retained prefix and recent window.
Applications that require arbitrary references into old context need another mechanism, such as a larger cache, selective retention, external retrieval, or an architecture designed for that access pattern. Sink preservation only addresses the specific disruption caused by removing sink positions.
Position handling remains part of the cache contract
Keeping selected KV entries is only valid when the runtime preserves the positional semantics expected by the model. Rotary, absolute, relative, and other position schemes impose different constraints on cached states and position indices. A cache implementation cannot assume that physically compacting entries is semantically identical to their original positions.
The sink policy therefore belongs beside position management in the serving design. Which entries remain, which position identifiers are supplied for new tokens, and whether cached keys require any transformation are implementation-specific questions.
This is also separate from extending a model beyond a supported context regime. Retaining sink tokens can make a bounded streaming cache behave better for a compatible model, but it does not by itself establish that arbitrary position lengths are valid.
The sink set is a model property, not a constant
A serving stack should not hard-code a universal sink count from another model. The number and location of high-attention early positions can vary with architecture, training setup, attention head, and checkpoint. A policy that retains four initial tokens is a configuration choice, not a mathematical guarantee.
Measurement should compare the target checkpoint under the exact cache policy intended for deployment. Useful inspection includes attention patterns where available, output quality under long streams, and sensitivity to changing the retained prefix size. The comparison also needs the same position-handling rules; otherwise cache eviction and position changes become confounded.
The practical boundary is precise: naive oldest-first KV eviction is unsafe for a model when positions being removed continue to play a structural role in its attention computation. Retaining a small prefix can preserve that role while bounding memory, but it cannot recover evicted content or certify unlimited context behavior.