A bounded KV cache seems to invite a simple eviction rule: keep the newest tokens and discard the oldest ones. For some transformer language models, that rule can degrade generation even when the discarded prefix carries little obvious semantic value. A small set of early positions may attract substantial attention across later decoding steps. These positions are commonly called attention sinks.
This behavior matters for streaming inference because cache eviction changes the attention computation itself. A fixed-size cache that preserves a few sink positions plus a recent window can behave differently from a cache containing only the same number of recent positions.
Attention normalization makes eviction observable
In causal self-attention, a query at the current position scores keys from positions that remain visible. Softmax then normalizes those scores across the visible set. Removing cached keys does more than delete their associated values: it changes the set over which attention mass is distributed.
For one attention head, the weights can be written as:
a_i = exp(s_i) / sum_j exp(s_j)where s_i is the score for visible key i. If an old key with a large score is removed, every surviving weight is renormalized. The resulting attention output can shift even when the evicted token would not appear relevant under a semantic reading of the prompt.
An attention sink is a position that repeatedly receives disproportionate attention mass. Early tokens can take this role in some trained models. The behavior is model-dependent, so a serving system should not assume that every architecture, checkpoint, layer, or head has the same sink pattern.
A recent-only cache changes more than context length
A sliding cache of width W retains the most recent W key-value pairs. Once generation exceeds that width, the oldest entry is removed for each new entry. Memory use stays bounded, but the model no longer receives the same attention state as full-cache decoding.
That distinction is easy to miss when cache size is described as context capacity. A model trained with full causal attention may depend on positions that a recent-only policy removes. The resulting mismatch is not equivalent to presenting the model with an independently constructed shorter prompt, because positional encoding and the retained KV states still reflect the longer generation history.
A sink-aware policy instead reserves cache slots for selected early positions and uses the remaining slots for recent tokens. With a total cache budget B, retaining S sink positions leaves roughly B - S slots for the moving recent window. Memory remains bounded while the attention computation keeps positions that the model may use as stable destinations for attention mass.
Sink retention does not preserve arbitrary long-range information
Keeping sink positions solves a narrower problem than general long-context retention. Sink tokens can stabilize attention behavior, but they do not contain every fact from evicted text.
Suppose a generated sequence contains a variable definition far from both the initial sink region and the recent window. Once its KV state is evicted, later tokens cannot attend directly to that position under ordinary cached self-attention. Retaining the first few tokens does not restore that dependency.
This creates two separate concerns. Sink retention can reduce degradation caused by removing structurally significant early positions. Long-range retrieval requires a policy or architecture that also preserves information-bearing positions outside the recent window. Treating these as the same problem can produce a cache that generates fluent text yet fails on dependencies spanning evicted regions.
Position handling must match the model
KV eviction interacts with positional encoding. Cached keys were produced at particular sequence positions, and a serving implementation cannot safely treat eviction as permission to renumber those states arbitrarily.
For models using rotary position embeddings, position information is incorporated into query and key representations. A cache implementation therefore needs a position strategy consistent with the model and inference method. Some streaming schemes use position transformations or cache-specific indexing rules; others retain the original logical positions. These are implementation choices with model-specific consequences, not interchangeable bookkeeping details.
The same caution applies to absolute position embeddings and other positional mechanisms. A cache policy that works for one model family does not establish equivalent behavior for another.
Cache size and effective dependency span are different quantities
A sink-aware cache can have constant storage while generation continues for far more tokens than the cache holds. That does not make the model’s effective attention span unbounded.
At any decoding step, direct attention is limited to the KV states still present. The model can carry some earlier information forward indirectly through recent hidden states, but that is not the same as retaining direct access to every prior token. Claims about streaming length should therefore distinguish bounded memory, stable generation, and preservation of long-range dependencies.
This distinction also affects evaluation. Perplexity or local continuation quality can expose some cache-policy damage, while tasks requiring references to distant content test a different capability. A cache design intended for open-ended generation and one intended for long-document question answering can require different evidence.
Full-cache decoding provides a useful reference
A practical evaluation compares bounded-cache output against the same model using an unpruned KV cache under matched decoding conditions. The comparison can isolate the effect of eviction from changes in sampling, prompt formatting, quantization, or model weights.
Useful measurements depend on the application. Token-level logit divergence can reveal local changes before sampled text visibly separates. Task metrics can expose lost long-range dependencies. Memory measurements confirm the actual cache bound. Inspecting attention patterns can support diagnosis, but high attention weight alone does not establish that a token is causally required for a particular output.
The cache budget should also be varied. Reserving sink slots reduces space available to recent context, so an excessive reserved region can trade one source of degradation for another. The relevant setting is the smallest retained sink region that maintains the target behavior for the specific model and workload.
Attention sinks are a serving constraint, not a universal token rule
Attention sinks are useful because they expose a hidden assumption in naive KV eviction: old positions are not necessarily interchangeable. Some positions can matter to the numerical structure of attention even when their text does not look informative.
That does not justify hard-coding a universal number of prefix tokens across models. Sink behavior should be verified on the checkpoint being served, under the positional and cache implementation actually used. Once that behavior is established, sink-aware retention can bound KV memory without confusing recent-token storage with preservation of the model’s full attention state.