Autoregressive language models normally reuse the keys and values of earlier tokens while generating the next token. This KV cache avoids recomputing the entire prefix at every decoding step, but its memory use grows with the cached sequence. A long-running chat, agent, or stream can therefore accumulate more cached state than a serving system wants to keep.
A tempting fix is a sliding window: retain only the most recent tokens and evict everything older. For models trained with ordinary dense attention, however, abruptly dropping all early tokens can damage generation quality even when those old tokens do not appear semantically important.
One reason is the attention sink phenomenon. Some transformer language models assign disproportionate attention to tokens near the beginning of a sequence. Keeping a small set of those initial tokens alongside the recent window can make bounded-cache streaming substantially more stable than keeping recent tokens alone.
This article explains the mental model, shows how the cache policy works, and draws an important boundary: attention sinks can help preserve local streaming behavior, but they do not give a bounded cache perfect memory of arbitrarily old content.
Start with the KV cache problem
In causal self-attention, a newly generated token can attend to earlier allowed positions. During decoding, each transformer layer has already computed key and value vectors for those earlier tokens. A serving system can cache those vectors and reuse them.
Conceptually:
prompt: A B C
cached: K_A V_A, K_B V_B, K_C V_C
next token D
compute K_D V_D
attend using cached earlier states + current stateAs generation continues, the cache grows:
3 tokens -> 3 cached positions per layer
1,000 tokens -> 1,000 cached positions per layer
10,000 tokens -> 10,000 cached positions per layerThe exact memory cost depends on the model architecture, numeric representation, batch layout, and serving implementation. The general pressure is straightforward: retaining more token positions requires more KV state.
If an application is truly streaming and only needs bounded recent context, limiting the cache can be more appropriate than allowing it to grow without bound.
Why a plain sliding window is not always enough
Suppose the cache budget is six positions. A simple recent-token policy behaves like this:
sequence positions: 1 2 3 4 5 6
cache: 1 2 3 4 5 6
new position 7 arrives
cache: 2 3 4 5 6 7
new position 8 arrives
cache: 3 4 5 6 7 8This looks reasonable if recent tokens contain the information needed for the next prediction. But it also changes the attention environment seen by the model. Tokens that were present at the beginning disappear entirely.
For some transformer language models, early positions receive substantial attention even when their token content is not obviously relevant to the current text. These positions act as attention sinks: they absorb attention mass that the model has learned to place there.
That behavior matters because softmax attention distributes probability mass across the keys that are available. Removing keys is therefore not equivalent to leaving them present but irrelevant. Once the initial positions vanish, attention is renormalized over a different set of keys, and the resulting layer activations can change.
The practical lesson is subtle:
old token has little semantic valuedoes not necessarily imply:
its cached position can be removed with no effectA cache policy must account for how the model actually uses positions, not only for what a human considers important text.
Keep sink tokens and recent tokens
A sink-aware streaming cache reserves two regions:
- a small fixed group of initial token positions; and
- a rolling window of the most recent positions.
With a six-position budget, imagine reserving two positions for sinks and four for recent tokens:
positions seen: 1 2 3 4 5 6
cache: 1 2 | 3 4 5 6
sink recent
position 7 arrives
cache: 1 2 | 4 5 6 7
position 8 arrives
cache: 1 2 | 5 6 7 8The total cache remains bounded. Positions 1 and 2 stay available, while the recent region advances with the stream.
This is the core mechanism behind sink-aware streaming methods such as StreamingLLM. The important point is not that the first words contain a permanent summary of the conversation. They generally do not. Their retained KV states preserve positions that the model’s attention mechanism may rely on as sinks.
That distinction prevents a common misunderstanding: sink tokens are structural anchors for attention behavior, not a compressed memory of all evicted content.
Why removing a key changes softmax attention
A small numerical example makes the effect easier to see.
Assume one attention head produces these unnormalized scores for three cached positions:
initial token: 4
recent token A: 2
recent token B: 1Softmax converts scores into normalized weights. Ignoring rounding, the exponentials are:
exp(4) = 54.60
exp(2) = 7.39
exp(1) = 2.72
sum = 64.71So the attention weights are approximately:
initial token: 0.844
recent token A: 0.114
recent token B: 0.042Now remove the initial key. The remaining scores have not changed, but normalization has:
recent token A: 7.39 / (7.39 + 2.72) = 0.731
recent token B: 2.72 / (7.39 + 2.72) = 0.269The two recent values now contribute in very different proportions to the head output simply because another key disappeared.
This toy calculation is not a model of every attention head, and a real transformer has many layers and heads. It demonstrates the mechanism that makes eviction non-neutral: softmax depends on the complete set of available scores.
Do not confuse streaming stability with long-term recall
Suppose a conversation begins with:
My deployment region is ap-southeast-1.Thousands of tokens later, that sentence has been evicted from a bounded recent window. Keeping the first few sink positions does not magically preserve the deployment region unless that information remains represented in retained state in a way the model can use.
A sink-aware cache therefore supports a different goal from full long-context attention:
bounded streaming cache
-> preserve useful attention behavior while old positions are evicted
full retained context
-> keep old token states available for later retrieval by attentionIf the application must answer questions about arbitrary details from far earlier in the stream, a bounded sink-plus-recent cache alone is the wrong abstraction. Options may include retaining a larger context, retrieving old information from external storage, maintaining an explicit application state, or using a model and serving strategy designed for the required long-context behavior.
This boundary is especially important for agents. Tool results, user constraints, identifiers, and decisions should not be assumed to survive indefinitely merely because generation remains fluent.
Treat the number of sink tokens as a model-dependent parameter
There is no universal sink count that is correct for every model.
The useful number depends on the model’s learned attention patterns and on the inference implementation. A configuration that works for one model family can degrade another. The same caution applies to the recent-window size.
Evaluate candidate settings on the actual model and workload. At minimum, compare:
full cache baseline
recent-only bounded cache
sink + recent bounded cacheMeasure more than whether generated text looks fluent. Depending on the application, useful checks include language-model loss on long streams, task accuracy, instruction retention, generation consistency, peak KV-cache memory, tokens per second, and latency distributions.
The full-cache run establishes the quality reference. The recent-only run shows whether naive eviction already works for the workload. The sink-aware run shows whether retaining initial positions recovers enough quality to justify the reserved cache space.
Account for positional encoding and cache bookkeeping
KV eviction is not only a list operation. Transformer implementations also track position information, and the correct handling depends on the model’s positional encoding and serving stack.
A model may use rotary position embeddings, learned absolute positions, relative schemes, or another mechanism. Cached keys can already contain transformations derived from their positions. An implementation that evicts entries and then incorrectly reassigns or recomputes positions can change model behavior independently of the intended cache policy.
For that reason, use an inference implementation that explicitly supports the target model’s streaming or cache-eviction strategy. Do not assume that deleting old KV tensors and renumbering the survivors is mathematically equivalent to the supported algorithm.
This is also why sink-aware streaming is an inference technique with model-specific compatibility requirements, not an API guarantee shared by all language models.
Understand the quality, memory, and latency trade-off
A bounded cache places a hard ceiling on the number of retained KV positions per sequence after the initial fill. That can make memory consumption predictable for long-running streams.
A smaller cache can also reduce the amount of attention work performed for each newly decoded token when the implementation actually attends only to retained positions. But end-to-end latency depends on more than attention arithmetic: batching, memory movement, kernels, scheduler behavior, hardware utilization, and other model operations all contribute.
The trade-off is therefore:
smaller retained cache
-> lower KV memory requirement
-> potentially less per-token attention work
-> less access to old token-level informationSink tokens improve one part of that trade-off by making aggressive eviction more stable for compatible models. They do not eliminate the information loss caused by eviction.
For short requests, the technique may provide little value because the cache never becomes large enough to create meaningful pressure. For workloads that require faithful use of the entire prompt, full-context or retrieval-oriented approaches may be more appropriate.
Common mistakes
Treating the first tokens as a semantic summary
Attention sinks are defined by attention behavior, not by their ability to summarize the sequence. Do not use them as a substitute for application memory or retrieval.
Assuming recent-only eviction is harmless
A sliding window is simple, but models trained with dense attention may depend on early positions in ways that are not obvious from token semantics. Benchmark it rather than assuming equivalence.
Copying one sink count across models
The phenomenon and useful configuration are model-dependent. Validate the number of retained initial positions for the exact model and serving implementation.
Measuring only memory
A cache policy can meet a memory target while silently reducing task quality. Compare quality against a full-cache baseline and test long streams that resemble production behavior.
Claiming infinite context
Bounded streaming can process an indefinitely continuing stream without retaining every old KV entry, but that does not mean the model has lossless access to an infinite history. Evicted information is unavailable unless some other mechanism preserves it.
When sink-aware streaming is a good fit
Consider this technique when all of the following are broadly true:
- generation can continue for much longer than the desired KV-cache budget;
- recent context is more important than exact recall of arbitrary old details;
- the target model exhibits compatible attention-sink behavior;
- bounded memory is operationally valuable; and
- evaluation shows acceptable quality relative to full-cache inference.
It is less attractive when requests are short, when every old token may become important later, or when the serving platform already provides a different long-context strategy that better matches the workload.
The simplest approach should win when it meets the requirement. If normal full-cache inference fits comfortably in memory and latency budgets, adding eviction policy, model-specific tuning, and new regression tests may not be worth the complexity.
Conclusion
A KV cache grows because autoregressive transformers retain earlier keys and values for future attention. Keeping only recent entries bounds that growth, but naive eviction can disturb models that rely on initial positions as attention sinks.
Retaining a small fixed set of initial KV entries alongside a rolling recent window provides a practical middle ground for compatible streaming workloads. It can stabilize bounded-cache inference without pretending to preserve the entire history.
Use the technique with the right mental model: sink tokens preserve attention behavior, not forgotten facts. Benchmark against full-cache and recent-only baselines, respect the model’s positional encoding and serving implementation, and choose a different memory strategy when the application genuinely needs long-term recall.