Autoregressive decoding can reuse key and value tensors from earlier token positions instead of recomputing them at every step. A conventional KV cache therefore grows as generation advances. A sliding-window cache places a bound on that retained state by keeping only a recent region.
The memory bound changes more than allocation size. Once an old key-value entry is evicted, a later attention operation cannot directly address that position through the cache. The resulting behavior depends on the model’s attention pattern, positional scheme, cache implementation, and any layers that use a different attention span.
KV state represents past positions used by attention
For one attention layer, a token at position t produces a key and value that can be represented as
k_t = x_t W_K
v_t = x_t W_VDuring a later decoding step, the new query can attend to cached keys and combine the corresponding values. With an unbounded causal cache, the retained state after T positions contains entries for the prior positions permitted by the model’s attention mask.
The cache avoids repeating key and value projection for those prior positions. It does not make attention independent of context length: storing more positions consumes more cache memory, and attending over more retained keys can also change the amount of work performed by the attention kernel.
A sliding window changes the retained set. For a window of width W, an implementation can keep a bounded recent interval rather than every key-value pair from the start of the sequence. The exact boundary and indexing convention are implementation details, so reproducing behavior requires the serving stack’s actual cache semantics.
Eviction removes direct attention targets
Suppose a later query would assign nonzero attention weight to a token far outside the retained window. If that token’s key and value have already been removed, the attention operation cannot include that position as a direct target. This is different from merely assigning it a small weight: the position is absent from the candidate set used by that operation.
That distinction matters when reasoning about long prompts. A nominal request length can exceed the number of positions retained by a local cache policy. The model can still carry information forward through hidden states produced at intermediate positions, but this is not equivalent to retaining the original token as a directly addressable key-value entry.
Information propagated through later representations may be transformed, compressed, mixed with other content, or omitted. Cache eviction therefore defines an access boundary, not a claim that all earlier information has vanished from every downstream representation.
A bounded cache does not imply identical behavior across layers
Transformer variants do not all use one uniform attention span. Some architectures combine local attention with layers that use a broader span. Others can apply different cache policies or attention masks at different layers.
As a result, a single number reported as the window size does not fully specify the model’s effective access pattern. If one layer retains a broad context while another uses a local window, the two layers expose different sets of prior positions to their queries.
Serving code can add another distinction. A cache object may have a fixed storage capacity while the attention mask defines a smaller usable region, or storage may be organized as a ring buffer whose physical slots are reused as logical positions advance. Physical buffer location and logical token position must not be treated as the same coordinate.
Position handling must survive slot reuse
A ring-buffer implementation can overwrite old slots while continuing to generate tokens with increasing logical positions. Attention code then needs enough position information to distinguish a newly stored token from the old token that previously occupied the same physical slot.
This is especially relevant when positional information participates in key and query construction. Reusing storage does not reset the sequence position by itself. The cache manager, attention kernel, and positional mechanism must agree on the mapping between logical positions and retained entries.
A bug in that mapping can produce behavior that resembles context corruption even when the tensor shapes and memory bounds look correct. Cache capacity tests alone are therefore insufficient; position progression and eviction boundaries also belong to the serving contract.
Prefill and decode can exercise different cache paths
Prompt prefill often processes many tokens together, while decode usually appends one or a small number of positions per iteration. An implementation can use different kernels or cache update paths for these phases.
With a sliding window, the transition matters when the prompt already exceeds the retained span. The serving stack must define which prompt positions populate the cache at the end of prefill and how the first decode step interprets their logical positions. A system that retains the most recent W entries after prefill is not equivalent to one that materializes a different subset and applies masking later.
Chunked prefill introduces another boundary. Processing a long prompt in chunks must preserve the same intended logical cache state across chunk boundaries. Chunk size is an execution detail only when the implementation maintains equivalent attention visibility and position semantics.
Cache size and model context limit are separate constraints
A model can expose a context limit that is larger than a local attention window. Those quantities describe different boundaries. The context limit constrains positions accepted by the model or serving configuration; the sliding window constrains which retained positions a particular attention operation can directly use.
Conversely, allocating a cache with capacity for many positions does not grant the model a broader supported context. Positional behavior, attention masks, architecture, and serving validation can impose limits independent of available memory.
For deployment work, the useful contract is therefore not just a cache byte count. It includes the retained logical position range, per-layer attention span, eviction rule, position mapping, prefill behavior, and decode update semantics. Once those details are explicit, memory budgeting and model-behavior analysis refer to the same cache policy rather than two different notions of context.