A decoder can keep its KV cache bounded by discarding states that fall outside a fixed recent-token window. The memory bound is attractive, but naive eviction can sharply degrade generation even when the removed tokens carry little obvious semantic value. A small set of key-value states from the beginning of the sequence can change that behavior.
This effect is associated with attention sinks: initial positions that receive substantial attention mass even when their token content is not important to the current prediction. StreamingLLM reported that preserving those initial states together with a rolling window of recent states recovers stable language-model behavior that ordinary window eviction can lose.
The mechanism is narrower than long-context retrieval. Sink tokens do not restore arbitrary information that has already been evicted. They preserve positions that the model’s attention computation has come to use as persistent destinations.
Naive window eviction changes the attention set
In causal self-attention, a new query attends to key-value states retained from earlier positions. A conventional full cache grows with the generated sequence because every prior position remains available.
A bounded cache can instead retain only the newest W positions:
cache_t = KV[t-W+1 ... t]Once the sequence exceeds W, the oldest entry is removed for every new entry added. Memory associated with cached sequence length then stops growing.
The computational change is not only memory management. Eviction changes the keys participating in softmax attention. If a model places persistent attention mass on early positions, deleting all of them forces that mass to be redistributed among the remaining keys. The resulting attention pattern can differ substantially from the pattern produced with the original prefix present.
Sink positions can matter without carrying current semantics
The StreamingLLM experiments identified strong attention toward initial tokens across several autoregressive transformer families. The paper describes these positions as sinks because they absorb attention scores even when they are not semantically informative for the current token.
That distinction matters. A sink is not equivalent to a summary token containing compressed history. Keeping a sink state does not preserve the content of discarded middle tokens. Its role is tied to the attention computation itself.
Softmax normalizes attention weights over the available keys. A key that consistently receives otherwise-unused attention mass can become structurally important to the distribution. Removing it alters the normalization domain and can redirect probability mass to keys that previously received less attention.
Later empirical work has also characterized sink behavior as closely related to key-side attention bias and softmax normalization. Those observations are model behavior, not a universal architectural guarantee for every attention variant.
Streaming keeps two regions instead of one
The bounded layout used by StreamingLLM retains a small initial region and a recent rolling region. Abstractly, with S sink positions and W recent positions:
retained KV = KV[0 ... S-1] + KV[t-W+1 ... t]The initial region remains fixed. The recent region advances as decoding proceeds. Tokens between those regions can leave the cache permanently.
This layout bounds the number of retained KV states by approximately S + W, independent of total stream length. It therefore separates sequence duration from resident KV-cache length.
The trade-off is equally explicit: once a non-sink token leaves the recent window, ordinary attention can no longer retrieve its key or value. A model may continue producing locally coherent text while losing direct access to facts or instructions that existed only in evicted positions.
Position handling remains part of the implementation
KV eviction cannot be treated as arbitrary array compaction when positional encoding affects attention. Cached keys were produced for particular token positions, and the model’s positional scheme determines how those states interact with later queries.
Streaming implementations therefore need position handling compatible with the model being served. Reindexing retained entries as though they had always occupied different positions can change attention scores. Rotary position embeddings are especially relevant because rotation is applied according to position; cache-management code has to preserve or correctly transform the positional relationship expected by the implementation.
This is an implementation boundary rather than a promise that one cache layout works unchanged for every transformer. Model architecture, positional encoding, attention kernel, and serving framework all participate in the result.
Bounded cache does not create an infinite context window
A streaming cache can operate for a stream far longer than its resident window, but that does not make every earlier token accessible. The distinction is between operational sequence length and active attention context.
With sink-plus-window retention, active state contains the fixed sink region and recent tokens. Middle history that has been evicted is absent unless another mechanism stores, summarizes, or retrieves it.
This prevents an important category error. Stable generation over a very long stream is not the same property as exact recall over that entire stream. A system that needs durable facts from distant history requires an additional memory mechanism or a model architecture designed for that access pattern.
Dedicated sink tokens move the role into pretraining
StreamingLLM also reported that adding a placeholder sink token during pretraining can concentrate the sink role in a designated position. That result indicates that sink behavior can be shaped by training setup rather than treated only as an accidental property of ordinary beginning tokens.
It does not imply that an arbitrary token inserted at inference time will acquire the same behavior. The dedicated-token result depends on the model having been trained with that arrangement.
For existing models, the practical observation is more conservative: measure which initial positions exhibit sink behavior, retain the required states, and evaluate the resulting cache policy on the target model and workload.
The useful boundary is attention stability, not memory recall
Attention sinks solve a specific failure mode of naive windowed KV eviction. They provide persistent early keys that can preserve a model’s expected attention distribution while the recent-token region moves forward. The cache can remain bounded without forcing every step to recompute the full prefix.
The mechanism does not recover evicted content, extend a model’s trained semantic context without limit, or guarantee identical behavior to full attention. Its value is the narrower systems property: a small persistent prefix can make bounded-cache streaming stable for models whose attention has developed sink positions.