Sliding-window attention imposes a finite token-distance boundary on causal attention. At position (t), a query can address only a recent interval of key and value positions rather than the full prefix. Once a cached position falls permanently outside that interval, later queries governed by the same local rule cannot address it.
That boundary changes the state required for autoregressive decoding. Full causal attention keeps usable KV state growing with sequence length. A fixed local window can keep the active KV span bounded, provided the runtime evicts or overwrites entries that have become unreachable.
The attention mask creates the retention boundary
For a causal window of width (W), one common indexing convention allows the query at position (t) to attend to positions (j) satisfying
[ \max(0, t-W+1) \le j \le t. ]
The exact endpoint convention varies across model definitions and kernels, so an implementation can differ by one position while still representing the same general local-attention structure. The architectural property is a maximum backward reach rather than a particular array slice.
With full causal attention, the eligible set at position (t) contains the entire prefix:
[ 0 \le j \le t. ]
The distinction is structural. In the local case, increasing (t) moves the left edge of the eligible interval forward. Old positions cease to be candidates for that layer’s attention.
Cache liveness follows future addressability
A KV entry is useful to a future decoding step only while at least one future query can reference its position. Under a strict fixed window, position (j) eventually becomes unreachable as the current position advances.
If the local rule admits the most recent (W) positions including the current position, then after the decode frontier has advanced sufficiently far past (j), the key and value at (j) are dead state for that layer. Retaining them cannot change subsequent attention results under that mask.
This is stronger than saying old tokens receive low attention scores. A masked-out position is excluded from the attention domain before softmax. Its cached K and V tensors have no path into that attention operation.
The distinction matters for cache policy. Score-based eviction is an inference heuristic unless it is part of the model’s defined mechanism. Window-based eviction can instead follow directly from the model’s positional eligibility rule.
Active KV storage can stop growing with sequence length
Ignoring metadata, allocator granularity, padding, and implementation-specific layouts, a layer with (H_{kv}) key-value heads and per-head dimension (d) stores two vectors per cached token position. With element size (b) bytes, full-prefix KV storage scales approximately as
[ M_{\text{full}}(T) = 2 T H_{kv} d b, ]
for cached length (T).
If the layer needs at most (W) token positions, the active local cache can instead scale as
[ M_{\text{window}} \approx 2 W H_{kv} d b. ]
For (T > W), the active tensor payload no longer needs to grow in proportion to (T). This statement applies to the KV state for that local-attention layer. Total inference memory can still grow because of request metadata, output tokens, allocator behavior, other layers, global-attention state, or runtime bookkeeping.
The bound also does not imply that every implementation immediately releases physical memory. A runtime may reserve a larger arena, reuse fixed blocks, or keep stale entries in allocated storage while excluding them logically. Logical cache liveness and allocator residency are separate properties.
A ring buffer matches the moving interval
Because the eligible interval advances monotonically during ordinary autoregressive decoding, a fixed-capacity circular buffer can represent local KV state without shifting the full tensor at every token.
A simple physical slot mapping is
[ s(t) = t \bmod W. ]
When position (t) arrives, its KV data can overwrite the slot previously associated with an older position that is no longer addressable. The physical slot number alone is not a token position, however. The runtime still needs enough positional information to associate each slot with the correct logical position used by the attention computation.
This becomes especially important with positional mechanisms such as rotary position embeddings. Reusing a storage slot does not reset the logical token index. Cache compaction and position semantics are distinct operations.
Paged cache managers can implement the same retention property with blocks rather than individual circular slots. Once every token in an old block is unreachable, that block can return to a free pool. The data structure changes; the liveness criterion does not.
Local attention does not make the model globally memoryless
Discarding a layer’s old KV entries does not mean information from those tokens has vanished from the model’s computation in every sense. Hidden states at later positions were themselves produced from earlier eligible context. Information can therefore propagate forward through successive local interactions.
At the same time, a query cannot directly retrieve arbitrary old KV vectors once they lie outside its window. The model’s effective long-range behavior depends on its architecture, depth, attention pattern, training, and any layers or tokens with broader connectivity.
Architectures can also mix local and global attention. If some layers retain full-prefix attention while others use a fixed window, only the local layers receive the strict window-sized KV bound. The global layers still require state according to their own attention domain.
Prefill and decode expose different operational effects
During prefill, many token positions are processed together. A local mask reduces the set of key positions relevant to each query, but actual memory traffic and temporary storage depend on the attention kernel. A kernel designed for local attention can avoid materializing work for the entire causal matrix; a generic path may expose different intermediate behavior.
During token-by-token decode, the retention consequence is more direct. Each new query needs only the KV entries still inside its allowed backward range. A cache manager can retire older state as the frontier advances.
Neither property alone guarantees a fixed latency improvement. Kernel design, batch composition, memory layout, quantization, device characteristics, scheduling, and cache-management overhead all affect measured latency.
Window size is an architectural parameter, not a cache-only knob
Reducing the retained KV span below the model’s defined attention window changes the information available to the attention operation. It is not equivalent to a lossless cache optimization.
Conversely, retaining more KV entries than the mask permits does not extend the model’s attention range. Extra cached tensors remain inaccessible if the mask excludes their positions.
The safe eviction boundary therefore comes from the attention rule itself: state can leave the active cache after no valid future query can reference it. Sliding-window attention supplies such a boundary by construction, turning an ever-growing full-prefix KV requirement into a bounded active span for the layers that use the local window.