A bounded attention cache creates a specific failure mode in autoregressive transformers: removing every old token can disturb attention even when those tokens no longer carry useful task content. Some early positions can absorb attention mass across many later queries. If cache eviction removes them, generation quality can degrade more than their semantic value would suggest.
These positions are often called attention sinks. The practical implication is narrow but useful: a streaming cache can keep a small prefix of sink positions while rotating the rest of its capacity through recent tokens.
A sink is about attention allocation, not stored facts
For one attention head, a query at position t assigns weights to cached keys through a softmax over attention scores. The weights are non-negative and sum to one. A position can therefore receive some probability mass even when no past token is strongly relevant to the current query.
An attention sink is a position that repeatedly receives such mass across later queries. This behavior is distinct from semantic retrieval. A sink token does not need to contain a fact that later output depends on. Its role emerges from the model’s attention patterns.
That distinction matters when designing eviction. A policy based only on token age assumes that old positions become uniformly disposable. Sink behavior violates that assumption: a very old position can remain structurally useful while newer positions carry the actual recent context.
Sliding windows remove two things at once
A conventional sliding-window cache keeps the most recent W key-value pairs and evicts everything before them. Once generation passes the window boundary, the original prefix disappears.
This gives bounded memory, but it also changes the set of keys available to every later softmax. If early sink positions had been receiving persistent attention mass, their removal forces that mass to be redistributed among the surviving keys. Repeating the operation as the window advances can move inference away from the attention pattern seen with an intact prefix.
A sink-aware cache separates the prefix from the rotating region. With cache capacity C, it can reserve S entries for initial sink positions and use the remaining C - S entries for recent context:
cache = [sink prefix] + [most recent tokens]The scheme does not make the context unbounded. Tokens between the retained prefix and recent region are still discarded. The preserved prefix addresses attention allocation; the recent region supplies local context.
Positional handling remains part of the design
Keeping selected key-value pairs is not sufficient if positional state is handled inconsistently. Transformer variants encode position in different places, and cache implementations must preserve the positional assumptions used by the model.
With rotary position embeddings, for example, positional rotation affects queries and keys. A streaming implementation must account for the positions associated with cached keys when the visible cache is compacted or shifted. Treating retained key-value tensors as if they had been produced at arbitrary new positions can change attention scores.
This is separate from sink retention. One decision determines which entries remain in memory; another determines how their positions participate in attention. A correct bounded-cache design has to satisfy both.
Sink count is a model-dependent parameter
There is no universal prefix size that applies to every transformer. Attention patterns depend on architecture, parameters, tokenization, and the inference setup. Reserving too much cache for sink positions also reduces space available for recent context.
The useful measurement is therefore not simply whether early tokens receive attention. Inspect how generation behaves across sequences much longer than the cache capacity, especially after many eviction cycles. Compare a recent-only window with a cache of the same total capacity that reserves a small prefix. The comparison isolates the allocation choice without giving one configuration extra memory.
Per-head inspection can add context because attention behavior need not be uniform across heads or layers. Aggregate statistics can hide a small subset that consistently places mass on early positions.
Sink retention does not recover evicted context
A retained sink prefix is not a substitute for long-context memory. Once a middle token leaves the rotating region, its key and value are unavailable to ordinary attention. If later generation requires information from that token, sink retention cannot reconstruct it.
This sets a clear boundary for the mechanism. Attention sinks address degradation caused by removing structurally persistent positions from a bounded cache. They do not solve retrieval over arbitrarily old content, nor do they remove the need to evaluate positional behavior and cache capacity together.
For streaming inference, the relevant question is therefore not just how many tokens fit in memory. Cache eviction also selects the attention structure that survives. Preserving a small sink prefix can keep that structure closer to the model’s established behavior while the recent region continues to move forward.