Autoregressive decoding keeps past attention keys and values so each new token can reuse earlier projections instead of recomputing them. That KV cache grows with sequence length, layer count, batch size, and the number and width of cached key/value heads. Reducing its numeric precision can cut the bytes occupied by those cached tensors, but it also changes the values consumed by later attention operations.
This makes KV cache quantization different from compressing data that is only stored and restored losslessly. The quantized cache remains on the inference path. Each subsequent query can interact with approximated keys and values, so the relevant question is not only how many bytes are saved, but where quantization error enters attention and how the serving implementation contains it.
Keys and values affect different parts of attention
For one attention head, a simplified computation can be written as:
scores = (Q K^T) / sqrt(d)
weights = softmax(scores)
output = weights VIf cached keys are quantized, their reconstruction error changes the dot products used to form scores. Those score changes occur before softmax, so they can alter the relative attention weights assigned to cached positions.
If cached values are quantized, their reconstruction error enters after the attention weights have been computed, through the weighted sum over V. The two error paths are therefore not interchangeable even when keys and values use the same storage format.
A serving system can quantize both tensors, only one of them, or use different quantization parameters for each. The appropriate choice depends on the model, numeric format, calibration method, kernel support, and acceptable output deviation. A memory reduction alone does not establish that two schemes are behaviorally equivalent.
Scale granularity controls which values share a quantizer
Low-bit quantization maps a range of real values onto a smaller set of representable levels. A scale and, for some schemes, a zero point define that mapping. The granularity of those parameters determines which cache elements must share the same numeric range.
A single scale for a large tensor is compact, but an outlier can expand the represented range and leave fewer effective levels for values concentrated near zero. Finer granularity, such as separate parameters per channel or per group, can adapt to local ranges at the cost of extra metadata and more involved kernels.
The exact useful grouping is implementation-specific. Key and value distributions can differ across layers, heads, channels, token positions, and models. A quantizer that works acceptably under one grouping cannot be assumed to preserve the same behavior after its grouping axis or group size changes.
This is also a reason to keep storage precision distinct from arithmetic precision. A cache may be stored in a low-bit representation and dequantized into a wider type before or during attention. In another implementation, specialized kernels may consume the compressed representation more directly. The storage format alone does not specify the arithmetic path.
Outliers can dominate quantization range
Quantization error is strongly affected by the range that must be represented. When a group contains a small number of values with much larger magnitude than the rest, a scale chosen to cover those values can reduce resolution for the more densely populated region.
Clipping changes that balance by limiting the represented range, but clipped values then incur explicit saturation error. The result is a calibration choice rather than a free reduction in error. The suitable clipping rule depends on the cache distribution and the behavior that must be preserved.
Outlier handling can also motivate mixed strategies. Some implementations may retain selected values or recent tokens at higher precision while compressing the rest. Such designs should be treated as specific algorithms with their own cache layout and kernel requirements, not as a universal property of KV quantization.
Quantization can happen at different points in cache lifetime
A decoder appends new keys and values as tokens are processed. A serving implementation can quantize each new cache entry immediately, quantize blocks after they reach a chosen size, or keep a recent region in a wider type before converting older entries.
Those choices change both memory behavior and the number of conversions on the inference path. Immediate quantization bounds the precision of nearly the entire cache early, while a residual high-precision region leaves recent states untouched until they age out of that region.
Block-based schemes also introduce boundary conditions. A partially filled block may require temporary storage or separate metadata. If quantization parameters are computed from a block, the implementation must define when those parameters become fixed and whether earlier entries can be requantized. These are serving-system semantics, not details that can be inferred from a bit-width label.
Memory accounting includes metadata and residual regions
A nominal conversion from 16-bit cache elements to 4-bit elements suggests a fourfold reduction for the raw element payload. Real cache memory accounting can be less direct.
Quantization scales, zero points, packing alignment, block metadata, high-precision residual regions, allocator rounding, and kernel workspaces can all consume additional memory. The cache may also coexist with model weights, activations, request metadata, and temporary attention buffers.
For capacity planning, the useful quantity is the actual allocated memory attributable to the cache under the target serving configuration. Bit width remains a major term, but it is not the complete memory model.
The same distinction matters for throughput claims. A smaller cache can reduce memory traffic in some execution paths, yet quantization and dequantization add work and may depend on specialized kernels. Whether request throughput or token latency improves is an empirical property of a concrete hardware, kernel, batching, and model configuration.
Error accumulates through reuse, not through repeated requantization by necessity
A cached key or value can be read many times as later tokens are generated. That repeated use means one approximation can influence many later attention computations.
This does not imply that the same cache element must be quantized again at every decoding step. A typical design can quantize an entry once, store it, and repeatedly read or dequantize that stored representation. Repeated influence and repeated quantization are separate concepts.
Longer contexts increase the number of cached positions available to attention and extend the lifetime of early entries. They therefore change the conditions under which cache quantization is evaluated. Short-context checks alone may miss behavior that appears when many quantized positions participate in attention.
Evaluation has to match the serving path
A quantized-cache implementation should be evaluated with the same cache format, grouping, residual policy, attention kernel, decoding settings, and context regime intended for deployment. Replacing the cache with full precision during evaluation removes the mechanism being tested.
Output comparison can be examined at several levels. Numeric tests can compare reconstructed cache tensors or attention outputs on controlled inputs. Model-level tests can compare logits or task-relevant outputs under matched decoding conditions. Serving tests can measure actual cache allocation and latency on the target execution stack.
No single metric establishes equivalence for every application. Small logit changes may be irrelevant under one deterministic decision boundary and may change token selection under another decoding configuration. The acceptance criterion needs to reflect the behavior the application depends on.
KV cache quantization is therefore best treated as an inference representation change with model-visible consequences. Its value comes from reducing the storage cost of states that grow with active context, while its boundary is set by the attention error, metadata overhead, kernel support, and output tolerance of the concrete serving system.