https://nalar.dev/tags/large-language-models/ Recent content in Large Language Models on Nalar Hugo en-us Sat, 05 Sep 2026 00:00:00 +0700 https://nalar.dev/keep-attention-sinks-for-stable-streaming-llm-inference/ Sat, 05 Sep 2026 00:00:00 +0700 https://nalar.dev/keep-attention-sinks-for-stable-streaming-llm-inference/ <p>Autoregressive language models normally reuse the keys and values of earlier tokens while generating the next token. This <strong>KV cache</strong> avoids recomputing the entire prefix at every decoding step, but its memory use grows with the cached sequence. A long-running chat, agent, or stream can therefore accumulate more cached state than a serving system wants to keep.</p> <p>A tempting fix is a sliding window: retain only the most recent tokens and evict everything older. For models trained with ordinary dense attention, however, abruptly dropping all early tokens can damage generation quality even when those old tokens do not appear semantically important.</p>