Autoregressive transformer inference keeps past key and value tensors so each new token can attend to prior positions without recomputing the full prefix. That KV cache grows with sequence length, layer count, batch size, and the number of stored key-value heads. At long contexts, its memory footprint can become a direct limit on concurrent requests or usable context length.
Quantizing the KV cache reduces bytes per stored element. The resulting approximation is not equivalent to quantizing a passive data structure, however. Cached keys participate in attention score computation, while cached values are mixed according to the resulting attention weights. Error in those two tensors therefore enters the attention operation at different points.
Keys affect the distribution used to read the cache
For one attention head, a query vector q is compared with cached key vectors k_j. Ignoring masks for a moment, the score for position j is:
s_j = q · k_j / sqrt(d)If a stored key is reconstructed from a quantized representation as k_j + e_j, its score becomes:
s'_j = s_j + q · e_j / sqrt(d)The key error is projected onto the current query. An error component orthogonal to q contributes nothing to that score, while an aligned component changes it directly. The same cached key can therefore produce different score error for different future queries.
Those perturbed scores pass through softmax. A small score change does not map to a fixed attention-weight change because softmax depends on the complete score vector. When several positions have similar scores, perturbations can alter their relative weights. When one position is separated by a large score margin, the same numeric perturbation can have a smaller effect on the ranking.
This makes key quantization a distribution-shaping approximation. Its effect is not captured fully by reconstruction error measured on key tensors alone.
Values affect the content after attention weights are formed
The attention output is a weighted sum of cached values:
o = sum_j a_j v_jwhere a_j is the attention weight assigned to position j. If only the values are quantized and a reconstructed value is v_j + r_j, then, with the attention weights held fixed, the output perturbation is:
delta_o = sum_j a_j r_jValue error is therefore aggregated through the attention distribution. Errors at positions receiving little attention contribute less to that output than equal-sized errors at positions receiving high attention.
In a real quantized cache, keys and values can both be approximate. Key error first changes the weights, and those changed weights are then applied to approximate values. Treating a single cache reconstruction metric as a complete quality measure hides this distinction.
A useful evaluation can keep separate measurements for key reconstruction, value reconstruction, attention-score drift, and downstream task output. The tensor metrics diagnose the codec; the later metrics show how the approximation propagates through model computation.
Quantization granularity controls which ranges share a scale
Low-bit quantization maps a set of floating-point values onto a limited set of representable levels. The scale and any offset determine which source range those levels cover. The group of elements sharing those parameters is therefore a major design choice.
A single scale over a large tensor is compact in metadata but must cover the range of every element in that group. If a small number of large-magnitude elements expand the range, resolution around smaller values becomes coarser. Finer groups can adapt to local ranges at the cost of more scale metadata and more complicated packing or kernels.
Possible grouping axes include token positions, channels, heads, or fixed-size blocks. These choices are not interchangeable. A per-token scheme adapts as activation ranges change across positions. A per-channel scheme can adapt to channels whose magnitudes differ consistently. Block schemes sit between broad tensor-level scaling and very fine metadata.
The suitable granularity depends on the cache tensor statistics produced by the specific model and on the inference implementation. A quantizer should not assume that keys and values have matching range structure merely because they have related tensor shapes.
Outliers can dominate a low-bit range
Uniform quantization is especially sensitive to the range assigned to each group. Suppose most elements occupy a narrow interval but a few have much larger magnitude. Expanding the scale to include those outliers increases the spacing between adjacent quantized levels for the rest of the group.
Clipping can trade outlier error for finer resolution in the central range. That trade is model- and tensor-dependent. Clipping a key component can alter future attention scores through query alignment, while clipping a value component changes the content available for aggregation.
Another option is to preserve selected recent tokens or selected tensor components at higher precision while quantizing the rest. Such mixed representations add bookkeeping and kernel complexity, but they make the error budget explicit rather than forcing every cached element through one numeric format.
Any special treatment also needs stable cache semantics. If entries move between precision classes as the cache grows, the implementation must define when conversion occurs and ensure that position indexing and attention masks still refer to the same logical tokens.
Residual high-precision windows change the error profile
A common cache design can retain a recent window in the model’s working precision and quantize older entries once they leave that window. This avoids repeatedly quantizing a cache entry while it is still recent and bounds the amount of high-precision cache memory.
The approach creates two attention regions with different numeric error characteristics. Recent positions are represented at higher precision, while older positions carry quantization error. If a workload frequently depends on distant context, the older region remains active in attention and cannot be treated as cold storage merely because its tokens are old.
Window size is consequently both a memory parameter and an approximation parameter. Increasing it raises cache memory but delays quantization for more positions. Decreasing it saves more memory earlier but exposes a larger fraction of the active context to the low-bit representation.
Evaluation should include prompts where relevant evidence appears at different distances from the generated token. Otherwise a benchmark dominated by short-range dependencies can understate the effect of quantizing older cache entries.
Memory accounting includes metadata and temporary buffers
The headline storage reduction from replacing a high-precision element with a low-bit element is only part of the memory calculation. Quantized groups need scales and, for some schemes, offsets. Packing may introduce alignment constraints. Kernels can also require temporary dequantization or accumulation buffers.
The effective bytes per cached token should include those costs. For a cache with N quantized elements grouped into sets of G, a simple accounting model is:
cache_bytes = packed_data_bytes
+ number_of_groups * metadata_bytes_per_group
+ persistent_auxiliary_bytesTemporary workspace should be measured separately because it affects peak memory even if it is not retained per token. This distinction matters when the objective is higher request concurrency: a smaller persistent cache can still encounter a peak-memory limit if the execution path allocates large transient buffers.
Bandwidth can matter alongside capacity. Attention over a long cache reads many stored keys and values. A compact representation reduces bytes fetched from memory, but any gain depends on whether the kernel can consume the packed format without offsetting that reduction through expensive conversion or poor memory access patterns. Storage format and kernel design therefore need to be evaluated together.
Cache quality needs sequence-aware evaluation
A one-time tensor error measurement cannot represent error accumulation across autoregressive generation. Each generated token creates another cache entry, and the model’s token choice affects the context used for later steps. Once quantization changes a token selection, subsequent executions no longer share an identical prefix with the higher-precision reference.
Two evaluation modes answer different questions. Teacher-forced or fixed-prefix comparisons can isolate numeric drift for identical queries and cache contents. Free-running generation captures the combined effect on actual decoding trajectories, but divergence makes per-position tensor comparisons less directly interpretable after the outputs separate.
Both views are useful when selecting a cache format. Fixed-prefix tests expose attention-score and output perturbation under controlled inputs. End-task checks show whether those perturbations matter for the behavior the application depends on.
KV cache quantization is most predictable when keys and values are treated as separate numerical interfaces to attention rather than as one homogeneous memory block. The memory target sets the compression pressure, but grouping, clipping, precision windows, and kernel behavior determine where approximation enters the computation. A format that fits the memory budget still needs an error budget tied to attention behavior, not only to bytes saved.