KV Cache Quantization Perturbs Attention Through Stored Keys and Values

During autoregressive inference, previously computed keys and values are reused from the KV cache instead of being recomputed for every new token. Quantizing that cache changes more than its byte representation. The stored approximation becomes an input to later attention operations, so its error can alter both attention scores and the vectors combined by those scores.

This boundary differs from quantizing model weights. A weight tensor is reused across requests, while KV state is generated from the current sequence and grows with its cached length. Its numeric range can also vary across layers, heads, positions, and requests.

Key error acts before the softmax

For one attention head, a new query q is compared with cached keys K:

s = q K^T / sqrt(d)
a = softmax(s)
o = a V

If the stored keys are reconstructed as K + E_K, the score vector becomes:

s' = q (K + E_K)^T / sqrt(d)
   = s + q E_K^T / sqrt(d)

The key error therefore enters before softmax. Its effect is not determined by the absolute quantization error alone. The current query direction matters because the score perturbation contains the product q E_K^T.

Softmax then maps the perturbed scores to a new probability vector. A small score change can have little effect when competing logits are well separated, or it can change relative attention substantially when several scores are close. A fixed bound on key reconstruction error is therefore not equivalent to a fixed bound on attention-probability error.

Value error acts after attention weights are formed

Suppose the keys are exact but cached values are reconstructed as V + E_V. With unchanged attention weights a, the output becomes:

o' = a (V + E_V)
   = o + a E_V

Value error enters through the weighted sum rather than through score formation. Each cached value contributes according to its attention weight for the current query.

In a real quantized cache, key and value errors can occur together. The output then reflects both a changed weight vector and approximate values. Treating all KV error as one scalar reconstruction metric can hide this distinction. Two quantizers with similar tensor-level error can affect attention differently if one preserves keys more accurately and the other preserves values more accurately.

Quantization granularity defines which values share a scale

Low-bit representations usually map a group of floating-point values through a scale and, depending on the scheme, a zero point or another offset convention. The grouping rule determines which values must share quantization parameters.

A scale shared across a large region is cheap in metadata but must cover that region’s numeric range. Finer groups can adapt to local ranges, at the cost of more scale metadata and additional indexing or kernel work. For KV state, possible grouping axes include token positions, channels, heads, or blocks, but the exact choices are implementation-specific.

This makes the bit width incomplete as a cache-quality description. Two implementations can both store four-bit elements while using different group sizes, clipping rules, scale precision, and key/value treatment. Their memory layout and numeric behavior need not match.

Cache growth changes the serving trade

For a transformer layer with cached key and value tensors, cache storage grows with the number of retained sequence positions. Reducing bits per cached element can reduce the dominant tensor payload, but total memory also includes quantization metadata, allocator overhead, padding, and any higher-precision region retained by the implementation.

Some serving designs keep recent tokens in higher precision and quantize older cache blocks. That arrangement separates two concerns: recent state can avoid immediate quantize-dequantize handling, while older state occupies less cache storage. It does not remove quantization error from older positions when those positions are attended later.

The practical effect also depends on the serving path. Dequantization may be fused into an attention kernel, performed into temporary storage, or handled by another backend-specific path. A smaller cache does not by itself establish lower end-to-end latency; memory traffic, conversion work, kernel support, batch shape, and sequence length all participate.

Calibration must match the quantity being protected

A cache quantizer can be evaluated by reconstruction error, but attention behavior is the downstream quantity that consumes the reconstructed tensors. Key error can shift score ordering or score margins, while value error changes the vectors aggregated after weighting.

That does not make tensor error useless. It means the metric answers a narrower question. A serving evaluation that only reports cache size and reconstruction error can miss changes in generated-token behavior, attention computation, or task-level output.

The implementation boundary is consequently precise: KV cache quantization is a state-compression mechanism whose approximation remains active in future token computation. Its suitability depends on the quantization scheme, the attention implementation, the model’s numeric behavior, and the serving workload. Bit width alone cannot characterize that boundary.