A sliding-window KV cache seems mechanically simple: keep the most recent tokens, evict older key-value entries, and continue decoding within a fixed memory budget. The complication is that some transformer models place substantial attention mass on a small set of early positions even when those positions carry little direct semantic relevance to the current token.
Those positions are often called attention sinks. If a cache policy removes them while preserving only the newest tokens, the attention distribution seen during decoding can change abruptly. A bounded cache can therefore behave differently from full-context inference even when the evicted text appears unrelated to the current request.
The practical issue is not that every model needs its first tokens forever. It is that cache eviction interacts with the normalization inside attention, and that interaction can make a few early key-value pairs structurally useful.
Softmax must place its probability mass somewhere
For one attention head, a query vector q assigns a score to each cached key k_i. In the usual scaled dot-product form,
s_i = (q · k_i) / sqrt(d)
a_i = exp(s_i) / sum_j exp(s_j)The coefficients a_i are nonnegative and sum to one across the positions available to that query. A head cannot assign zero total mass to the cache. Even when no prior token is strongly relevant, softmax still distributes all of its mass among the keys that remain visible.
This creates room for a position to act as a sink. A key that is consistently available and receives a relatively favorable score can absorb probability mass that does not need to contribute much useful content through its associated value vector.
The sink behavior is therefore not equivalent to ordinary retrieval of an early fact. A token can matter because of its role in the attention distribution rather than because its text should influence the next-token prediction in an obvious semantic way.
That distinction matters for eviction. A recency policy reasons about which text is likely to remain useful. Attention also depends on the set of keys over which softmax is normalized.
Eviction changes both membership and normalization
Suppose an attention head sees keys with scores
[5.0, 1.2, 1.0, 0.8]and the first position is an attention sink. Removing that first key does more than delete one contribution. The remaining three coefficients are renormalized over a different denominator.
The output of the head is
output = sum_i a_i * v_iso both effects matter: the sink value disappears, and the weights on every surviving value increase relative to their previous normalized weights.
This is a general property of softmax attention. It does not imply that the first position will dominate every head, layer, prompt, or model. Sink behavior is model-dependent, and the magnitude can vary across heads and decoding positions.
It does imply that a cache policy cannot be evaluated solely by asking whether discarded text still contains useful facts. Eviction changes the numerical context in which all retained values are combined.
A pure sliding window can remove a stable anchor
A fixed sliding window of size W retains positions
t - W + 1 ... tat decoding position t. Once generation advances far enough, every early token leaves the cache.
A sink-aware cache reserves a small prefix and uses the rest of the budget for recent positions. If S prefix positions are retained, a conceptual layout is
[0 ... S-1] + [t-(W-S)+1 ... t]for a total budget near W, subject to the implementation’s indexing and current sequence length.
The reserved prefix does not preserve the entire old conversation. Tokens between the prefix and recent window are still evicted. The policy instead keeps two kinds of state: a small fixed region that can preserve sink behavior and a moving region that carries recent context.
This layout has a different purpose from retrieval-augmented context management. It does not select old tokens based on semantic similarity, nor does it reconstruct missing content. Its target is the behavior of local autoregressive inference under bounded KV-cache memory.
Position handling is part of the cache policy
Keeping the right key-value entries is not sufficient if positional treatment changes their meaning.
Transformer implementations can encode position through different mechanisms. With rotary position embeddings, for example, position affects query and key rotations before their dot product is formed. Cache eviction code must respect the positional assumptions of the specific model and attention implementation.
A naive implementation might compact surviving cache entries and treat them as if they had always occupied consecutive positions. That can change attention scores because the model’s positional representation no longer corresponds to the intended token relationships.
Some streaming designs adjust or recompute positional treatment for retained tokens; others rely on model-specific cache semantics. There is no universal rule that arbitrary KV tensors can be sliced, concatenated, and reassigned positions without affecting outputs.
For developers, this makes the cache interface significant. A framework may expose a cache object that tracks positions, sequence offsets, or rotary state in addition to raw key and value tensors. Bypassing that abstraction can produce a cache with the expected shape but different model behavior.
Sink retention does not recover evicted information
Retaining early sink positions addresses one failure mode of bounded attention. It does not make a short cache equivalent to an unlimited context window.
If a user states a constraint thousands of tokens ago and that token is neither in the retained prefix nor in the recent window, the model cannot attend to its original key-value entry. Keeping sink tokens does not encode the missing constraint into them.
The same boundary applies to long-range references, source passages, tool outputs, and intermediate reasoning state. A sink-aware window can preserve more stable attention behavior while still losing semantic information that falls outside the cache.
This separates two engineering questions that are easy to mix together:
- Does eviction disturb the model’s attention mechanics?
- Does eviction remove information needed for the current prediction?
Sink retention is aimed primarily at the first question. Context selection, summarization, external retrieval, or a larger effective window may be needed for the second.
The number of retained sink tokens is model-specific
There is no architecture-independent constant for the number of prefix tokens that should be pinned. A model may concentrate sink behavior in one position, several early positions, particular heads, or not strongly enough for special handling to matter under a given workload.
Retaining more prefix tokens also consumes space that could otherwise hold recent tokens. With a fixed cache budget, increasing S reduces the moving window from W to roughly W - S.
That makes S a policy parameter rather than a decorative constant. Its effect should be evaluated with the exact model, tokenizer, positional scheme, cache implementation, and sequence lengths used by the application.
The beginning-of-sequence representation also matters. Chat templates can insert system markers, role tokens, or other fixed prefixes before user text. The cache positions that behave as sinks are positions in the actual token sequence presented to the model, not necessarily the first visible words in an application transcript.
Evaluation needs sequences long enough to trigger eviction
A cache strategy can appear correct on short prompts because no token has been removed yet. Tests need to cross the eviction boundary and continue far enough for differences to become visible.
A useful comparison keeps model weights and decoding settings fixed while varying cache policy. Full-cache decoding provides a reference when the sequence fits within the model’s supported context. A pure recent-token window and a sink-aware window can then be compared against that reference.
Exact token equality is a strict signal but can be brittle once small numerical differences alter sampling. Logit or probability comparisons at matched prefixes can expose divergence before generated sequences branch. Greedy decoding removes sampling randomness, although it does not make two cache policies mathematically equivalent.
Memory accounting should be checked separately. A sink-aware policy is still bounded only if the implementation actually evicts middle entries instead of retaining hidden references to old tensors.
The relevant result is conditional: a chosen cache policy, on a specific model and sequence regime, preserves acceptable output behavior within a stated memory budget. The mechanism does not justify a universal quality guarantee.
Cache boundaries belong in the model contract
KV-cache eviction is often treated as an infrastructure optimization because it changes memory residency rather than model weights. Attention sinks show the limit of that view. The set of cached keys is an input to every subsequent attention computation, so changing that set can change the function executed by the model.
A production interface that supports bounded streaming inference should therefore make cache policy explicit. The window size, pinned-prefix size, positional handling, and eviction semantics are part of the inference configuration, just like decoding temperature or maximum context length.
This also makes upgrades easier to reason about. Replacing a model or attention backend can invalidate assumptions about sink positions or cache indexing even when the public generation API stays unchanged.
Attention sinks do not remove the fundamental information loss of a bounded window. They expose a narrower point: the oldest tokens can have a numerical role that recency alone does not capture. When a model exhibits that behavior, preserving a small prefix can keep the attention mechanism closer to its full-cache regime while the rest of the cache remains bounded.