A bounded KV cache can keep inference memory from growing with an unending token stream, but deleting every old token in strict arrival order can disrupt attention more than the missing content alone suggests. In some autoregressive transformers, early positions receive substantial attention even when their token semantics are not directly relevant to the current prediction. These positions act as attention sinks.
The serving consequence is specific: a sliding cache that preserves a small initial prefix alongside the most recent tokens can behave differently from a same-sized cache containing only the newest tokens. The mechanism concerns attention state and positional handling, not retrieval of forgotten text.
Softmax must place probability somewhere
For a query at one attention head, causal self-attention assigns normalized weights over visible keys:
scores = (q K^T) / sqrt(d)
weights = softmax(scores + mask)
output = weights VThe weights sum to one across positions that remain visible. A model can therefore develop patterns in which some positions absorb probability that is not strongly tied to useful token content. Empirical work on attention sinks has observed this behavior at initial token positions in several autoregressive transformer settings.
This is a model behavior, not a mathematical guarantee that the first token must become a sink. Its strength depends on architecture, training, position representation, layer, head, and input. A serving implementation should not infer a universal prefix size from the mechanism alone.
FIFO eviction removes both content and a familiar attention target
Consider a cache budget of six positions after a longer stream has arrived. Pure FIFO retention might contain only recent states:
... evicted ... | t19 t20 t21 t22 t23 t24A sink-aware layout can reserve part of the budget for an initial prefix:
t0 t1 | ... evicted ... | t21 t22 t23 t24Both layouts discard historical content. The second layout does not restore information from t2 through t20. It only preserves selected early key-value states while keeping a recent window.
That distinction prevents an incorrect interpretation of attention sinks as long-term memory. Facts present only in evicted middle tokens are no longer directly available through their removed KV entries. Prefix retention addresses a stability property of attention under cache truncation; it does not make a bounded cache semantically equivalent to full-context attention.
Position handling is part of cache correctness
KV eviction cannot be specified only as a list of tensors to retain. Attention also depends on the positional representation associated with those states.
With absolute position embeddings, cached states already reflect the positions used when they were produced. With rotary position embeddings, position-dependent rotations are applied to queries and keys. A runtime that shifts, reindexes, or reconstructs cached states must preserve the positional semantics expected by the model and its attention kernel.
This becomes especially relevant when a fixed prefix and a moving recent window coexist. Logical adjacency in the compact cache buffer does not imply that the retained tokens were adjacent in the original stream. Implementations can use different position-management strategies, so buffer indices must not be treated as model positions without checking the runtime contract.
A sink prefix consumes the same bounded budget
Keeping sink tokens has a direct capacity cost. If the cache limit is C, the retained prefix uses S positions, and no other reserved regions exist, at most C - S positions remain for the moving recent window.
A larger prefix therefore shortens recent-history coverage under a fixed cache budget. A smaller prefix leaves more recent states but may not preserve enough of the attention pattern for a given model. This is not a generic optimization with one correct constant.
The memory bound also includes the usual KV geometry: cached layers, KV heads, head dimension, element representation, batch or request layout, and allocator overhead. Sink retention changes which token positions occupy the cache; it does not remove those other terms.
Attention sinks and sliding-window attention are different mechanisms
A model with architectural sliding-window attention has an attention pattern defined by the model itself for relevant layers. A runtime that truncates the KV cache of a model originally built for broader causal attention is imposing a serving constraint after model computation has been defined.
Sink-aware cache retention can reduce a failure mode of that imposed truncation in compatible settings, but it does not convert full attention into native sliding-window attention. Likewise, a model that already has bounded local attention may have cache requirements determined by its architecture rather than by an external sink policy.
The same separation applies to KV cache quantization. Quantization changes the representation precision of retained states; sink-aware eviction changes which positions remain. They can coexist, but they solve different memory terms and introduce different error modes.
Validation must include long streaming sequences
A cache policy can appear correct on prompts shorter than its capacity because no eviction occurs. Its behavior changes only after the stream crosses the cache limit. Tests therefore need to exercise the eviction regime and compare outputs against an appropriate reference for the same model and positional configuration.
Exact token equality is not guaranteed even for cache methods intended to approximate full-context behavior. Useful evaluation can inspect output divergence, task behavior, numerical stability, cache occupancy, and latency under the actual runtime. The acceptable boundary depends on the application rather than on sink retention alone.
Attention sinks provide a concrete reason that KV eviction is not merely a storage queue. Once a runtime removes states from an attention context, token selection and positional semantics become part of model behavior. Retaining a small prefix can preserve a useful attention target while bounding memory, but the removed middle context remains removed, and that semantic limit should stay explicit in the serving design.