Stream Long LLM Sessions with Attention Sinks

Long-running LLM sessions create a simple resource problem: every generated token can add keys and values to the attention cache. Keep the entire history and memory use keeps growing. Keep only the newest tokens and some transformer models degrade sharply once older cache entries disappear.

Attention sinks provide a useful middle ground for compatible models. Instead of retaining the full KV cache, keep a small group of initial tokens plus a moving window of recent tokens. The cache stays bounded, yet the model can remain much more stable than with a recent-token window alone.

This article builds a practical mental model for attention sinks, shows how the cache policy works, and separates what this technique preserves from what it deliberately discards.

Start with the KV cache problem

During autoregressive generation, a transformer repeatedly attends to tokens that came before the next token. Recomputing every previous key and value at every step would waste substantial work, so inference systems commonly store those tensors in a KV cache.

Conceptually, a growing conversation looks like this:

step 1: cache [t1]
step 2: cache [t1 t2]
step 3: cache [t1 t2 t3]
...
step n: cache [t1 t2 ... tn]

The exact memory cost depends on the model architecture, numeric format, batch size, and cache representation. The important relationship is simpler: if every past token remains cached, cache memory grows with sequence length.

A natural fix is a sliding window. If the budget is four tokens, keep only the newest four:

before eviction: [t1 t2 t3 t4 t5]
after eviction:      [t2 t3 t4 t5]

This bounds memory. It also changes the attention context. Once t1 disappears, later queries can no longer attend to its cached key or value.

For some models, repeatedly dropping the oldest entry causes unexpectedly poor generation even when the evicted token carries little obvious semantic information. Attention sinks address that specific failure mode.

The mental model: keep an anchor and a moving window

Many softmax-attention language models assign substantial attention mass to a few tokens near the start of a sequence. These early positions can act as attention sinks: queries direct attention toward them even when their text is not important to the current meaning.

A streaming cache can preserve those initial positions and use the remaining budget for recent context. With two sink tokens and four recent tokens, the retained state might look like this:

full history:
[s1 s2 a b c d e f g h]

retained cache:
[s1 s2 | e f g h]

After another token arrives:

[s1 s2 | f g h i]

The first two entries stay fixed. The recent region moves forward.

This is different from treating the first tokens as a summary. The sink entries do not need to encode the discarded conversation in a human-readable or semantic sense. Their useful role comes from the model’s attention behavior. Retaining them preserves attention destinations that the model has come to rely on.

Softmax makes a sink useful

For one attention head, a query produces a score for each available key. Softmax converts those scores into non-negative weights that sum to one:

attention_weight_i = exp(score_i) / sum_j exp(score_j)

Every query must distribute its attention mass across the keys that remain visible. In trained transformer language models, some heads can place persistent mass on early positions. An initial token can therefore become a convenient destination for attention that does not need to contribute much useful value content.

If a serving system suddenly removes those positions, it changes the set over which attention is normalized. A plain sliding window can therefore disturb attention patterns beyond the obvious loss of old semantic content.

Keeping sink tokens does not make softmax attention independent of eviction. It preserves a small part of the original attention structure that can matter disproportionately for streaming stability.

The behavior is model-dependent. A serving system should not assume that every transformer, attention variant, or checkpoint has the same sink pattern.

A bounded cache policy

A simplified streaming policy needs two parameters:

sink_count   = number of initial positions kept permanently
recent_count = number of newest positions kept

The logical cache budget is then:

cache_budget = sink_count + recent_count

Suppose sink_count = 4 and recent_count = 2044. After the stream grows beyond 2048 retained positions, each new token can evict the oldest entry from the recent region while the first four positions remain.

Pseudo-code for the retention decision is small:

keep = first(sink_count) + last(recent_count)

A real implementation has more work to do. Key and value tensors are usually stored per layer, and positional encoding must remain consistent with the model and inference engine. Cache eviction cannot be implemented safely by treating token arrays as if position handling were irrelevant.

That distinction matters because the cache policy describes which states remain, not every tensor operation required to make a specific model produce correct attention scores.

Attention sinks do not preserve old facts

The most important limitation is easy to miss: a bounded streaming cache forgets discarded token states.

Consider a conversation that starts with:

User preference: send reports as CSV.

Thousands of tokens later, that sentence may fall outside the recent window. Keeping initial sink tokens does not guarantee that the model can still recover the CSV preference. The sink states are not a general-purpose compressed memory of everything evicted between the beginning and the current window.

This gives two separate goals:

• streaming stability: keep generation behavior usable as the stream exceeds the cache window;
• long-range recall: retrieve specific information from distant earlier content.

Attention sinks primarily target the first goal. Applications that require the second need another mechanism, such as explicit state, retrieval, periodic summaries, or a model and serving setup that retains the required long-range context.

This distinction prevents a common design error: interpreting an effectively unbounded stream duration as an effectively unbounded information horizon.

Compare three cache strategies

The trade-off becomes clearer by comparing full retention, a plain recent window, and a sink-aware window.

A full cache is the straightforward choice when the session fits the available memory and distant context matters. There is little reason to evict useful states merely to adopt a more complex policy.

A plain window can be appropriate for a model designed or validated for local attention behavior. Sink-aware retention becomes interesting when tests show that naive eviction damages generation and the target model exhibits the relevant sink behavior.

Choose the window from application needs

There is no universal sink count or recent-window size. The right values depend on the checkpoint, serving implementation, workload, and acceptable quality loss.

The recent window controls how much local text remains directly available. A larger window can preserve more nearby context but consumes more cache memory. A smaller window saves memory but makes information disappear sooner.

The sink count is usually small relative to the recent region. Increasing it consumes cache slots that could otherwise hold recent tokens, so retaining extra initial positions without evidence can be counterproductive under a fixed budget.

Treat both values as deployment parameters that require measurement. Useful evaluation streams should be much longer than the retained cache and should resemble production traffic. Short prompts cannot expose failures that appear only after repeated eviction.

Test more than token-level quality

A streaming configuration can look acceptable on aggregate language-model metrics and still fail an application. Evaluation should reflect what users expect the session to do.

For a long assistant conversation, useful checks can include:

• generation stability after many cache turnovers;
• adherence to instructions that remain inside the recent window;
• behavior immediately before and after an eviction boundary;
• latency and memory as stream length increases;
• explicit tests showing which distant facts are no longer recoverable.

The final item is especially important. If a product needs durable user preferences or task state, test those requirements separately instead of assuming the streaming cache supplies them.

Memory measurements should also use the actual serving stack. Model architecture, grouped-query attention, cache precision, allocator behavior, batching, and other implementation choices affect real memory use. A token-count budget is useful for reasoning, but it is not a substitute for profiling.

Common mistakes

The first mistake is describing attention sinks as semantic summaries. That creates false expectations about distant recall. Their role is tied to attention behavior, not to guaranteed storage of evicted content.

The second is applying the policy to an arbitrary model without validation. Sink behavior has been observed across many transformer language models, but architecture and training choices can change attention patterns. Measure the checkpoint you plan to serve.

The third is ignoring positional handling. Retaining non-contiguous cache regions can interact with positional encodings and cache indices. Use an implementation designed for the target architecture rather than deleting tensor slices and assuming the result is equivalent.

The fourth is evaluating only short sequences. A cache policy intended for long streams should be tested after the stream has exceeded the cache budget many times. Otherwise, the test barely exercises eviction.

The fifth is using streaming eviction when full context comfortably fits. Bounded caches trade retained information for resource control. If memory pressure is absent and old context is valuable, full retention is simpler and preserves more information.

Use attention sinks for the problem they solve

Attention sinks are useful when an LLM service must continue decoding across a long stream with bounded KV-cache memory and a plain sliding window destabilizes the target model. The practical pattern is compact: preserve a few initial cache entries, preserve a recent window, and evict the middle history as the stream advances.

That pattern solves a resource and attention-stability problem. It does not turn a fixed-size cache into perfect long-term memory.

A sound deployment therefore starts with two separate questions: how much cache can the service afford, and how much distant information must the application retain? Use sink-aware streaming for the first constraint when the model benefits from it. Add explicit memory or retrieval for the second when old facts must remain available.