https://nalar.dev/tags/long-context/ Recent content in Long Context on Nalar Hugo en-us Thu, 24 Sep 2026 00:00:00 +0000 https://nalar.dev/sliding-window-attention-bounds-each-query-to-a-local-token-horizon/ Thu, 24 Sep 2026 00:00:00 +0000 https://nalar.dev/sliding-window-attention-bounds-each-query-to-a-local-token-horizon/ <p>A causal attention layer does not always expose every earlier token to every query. With a sliding window of width <code>w</code>, the query at position <code>i</code> can be restricted to recent positions rather than the full prefix. The attention graph becomes local: old tokens fall outside the direct edge set even though they remain part of the sequence.</p> <p>That boundary changes computation, memory traffic, and information paths at the same time. It is not merely an optimized implementation of full attention. Once the mask removes distant key-value pairs, the layer implements a different dependency pattern.</p>