A causal attention head can assign noticeable probability mass to an early token even when that token is not a strong semantic match for the current query. The token then behaves as an attention sink: its key remains a convenient destination for probability that the head does not direct toward content-bearing positions.
This behavior matters during autoregressive serving because a KV-cache policy can preserve recent tokens yet still alter model behavior sharply if it removes sink positions. Recency alone does not describe the functional role of every cached key.
Softmax must distribute its full probability mass
For one attention head, a query at position t produces logits over permitted keys:
s_i = q_t^T k_i / sqrt(d)and softmax converts them into weights:
a_i = exp(s_i) / sum_j exp(s_j)The weights over unmasked positions sum to one. There is no separate output meaning “none of these keys is useful.” If a head benefits from keeping the contribution of some content positions small, assigning mass to another available position can serve as part of that behavior.
An early token is structurally available to many later queries in a causal decoder. That repeated availability can make an early position distinct from ordinary content tokens. The sink interpretation concerns the attention pattern and its effect on computation; it does not imply that the token’s text itself has special semantic relevance.
A sink is not equivalent to a high-value memory token
High attention weight is sometimes read as evidence that a token contains information needed by the current prediction. That interpretation is too strong for a sink position.
The output of a head is
o_t = sum_i a_i v_iso the effect of a large a_i also depends on the associated value vector v_i, downstream projection, residual stream, and later layers. A position can receive substantial attention while its role is dominated by the geometry of normalization rather than by retrieval of matching content.
This distinction also blocks a common inference error: an attention heat map alone does not establish semantic dependence. It shows the weights used by that head, not a complete causal account of the model’s output.
Cache eviction can break a stable normalization pattern
A sliding KV cache usually keeps a recent window and discards older keys and values. For ordinary local dependencies, that policy has an intuitive basis: distant tokens become unavailable while nearby context remains.
A sink changes the boundary. Suppose an early key has received persistent mass across many later positions. Removing it changes the denominator of softmax and forces probability to be redistributed among the keys that remain. Even if the evicted token carried little relevant text, the head no longer operates under the same normalization pattern.
Keeping selected sink positions alongside a rolling recent window can therefore preserve a computational role that pure recency eviction would remove. This is a cache-layout consideration, not a claim that every model, layer, or head uses the same sink positions.
The first token is a common structural candidate, not a universal rule
In a causal sequence, the first position is visible from every later position. That makes it a natural candidate for persistent attention. Special beginning-of-sequence tokens can also occupy this location, depending on the tokenizer and prompt format.
The structural argument does not establish that position zero must be a sink. Attention patterns depend on model parameters, architecture, positional representation, prompt construction, and head. Some heads can remain local, some can track content, and some can place mass on positions that act as sinks.
A serving implementation should therefore avoid encoding “the first token is always sufficient” as an architectural guarantee. If sink retention is used, the selected prefix length and eviction policy are properties to validate for the specific model and workload.
Sink retention does not restore discarded context
Preserving a few early KV entries does not make a bounded cache equivalent to the full context. Tokens removed from the middle or older portion of the sequence are still unavailable as keys and values. Any dependency on their content can still be lost.
The retained sink only addresses a narrower failure mode: disruption caused by deleting positions that participate disproportionately in the attention normalization pattern. It does not recover facts, instructions, entities, or long-range relations stored exclusively in evicted tokens.
This boundary is useful when evaluating streaming inference. A cache can remain numerically stable relative to a naive sliding window while still losing task-relevant distant context. Those are separate properties and require separate measurements.
Attention masks and position handling remain part of the cache contract
Retaining an old sink entry beside recent entries creates a non-contiguous logical cache. The attention implementation must preserve the intended causal relationships and positional convention for every retained key.
With position-dependent attention mechanisms, moving a cached entry to a different physical slot must not silently change its logical position unless the model’s cache scheme explicitly defines such a transformation. Physical storage order, logical token index, attention mask, and any position encoding used by query-key interaction need a consistent contract.
This is especially relevant when a runtime compacts cache memory. A tensor can be compact in storage while representing positions that are far apart in the original sequence. Treating the compacted tensor index as the original position can alter attention scores independently of the eviction decision itself.
Attention sinks expose a specific serving boundary: cache importance is not identical to token recency or semantic relevance. A small set of positions can matter because of the normalization behavior of attention. Retaining them can preserve that role, while the rest of the cache policy still determines which actual context remains available to the model.