Autoregressive transformer inference reuses attention keys and values from earlier tokens so each new token does not recompute the full prefix. That reuse creates the KV cache, whose memory grows with the number of cached tokens. At long context lengths or high request concurrency, the cache can become a major part of inference memory.

KV cache quantization changes the representation of those stored tensors. Keys and values are written in a lower-precision format together with any scale or metadata needed for reconstruction. Attention later consumes reconstructed values or uses a kernel that handles the quantized representation directly.

The memory reduction is concrete, but it is not free compression. Quantization introduces approximation error, metadata, conversion work, and implementation constraints. Those effects depend on the quantizer and on the numerical structure of keys and values.

Cache size grows with tokens, layers, and KV heads

For a decoder with a cache at each transformer layer, a simplified cache element count is

elements = 2 * layers * cached_tokens * kv_heads * head_dim

The factor of two represents keys and values. Multiplying by bytes per stored element gives the main tensor storage before allocator overhead and quantization metadata.

This expression also shows the dimensions that matter. Extending the context increases cache storage linearly in the number of retained tokens. Serving more sequences adds another request or batch dimension. Architectures using fewer key-value heads, such as grouped-query or multi-query attention, reduce the cache independently of numeric precision.

Quantization acts on the bytes associated with each cached element. It does not change the number of tokens or attention positions represented by the cache.

A scale defines the quantized coordinate system

A common low-bit representation maps a floating-point tensor into integer codes using a scale and, for asymmetric schemes, a zero point. A symmetric form can be written conceptually as

q = clamp(round(x / scale), q_min, q_max)
x_hat = q * scale

x_hat is the reconstructed approximation used by later computation. Values that fall between representable levels incur rounding error. Values outside the chosen numeric range can be clipped.

The scale can cover an entire tensor, a head, a channel, or a smaller group of elements. Coarser grouping stores less metadata but forces more values to share one range. Finer grouping can adapt to local magnitude differences at the cost of more scales and more indexing work.

The nominal bit width therefore does not fully describe the cache format. Two schemes with the same number of bits per code can differ in group size, scale precision, clipping policy, packing layout, and resulting error.

Keys and values can have different numeric structure

Keys and values serve different roles in attention. Keys participate in dot products with the current query before the softmax. Values are combined using the resulting attention weights.

An error in a cached key can perturb an attention logit. Because the softmax couples logits across positions, that perturbation can change the distribution of attention mass rather than only the contribution from one cached vector. An error in a value instead affects the vectors being aggregated after attention weights are formed.

This does not imply a universal bit width or quantizer for either tensor. It does mean that treating keys and values as interchangeable storage arrays can hide different error paths. Separate calibration rules, group axes, or precision choices can be justified when their observed ranges and downstream sensitivity differ.

Outliers can dominate a shared quantization range

Suppose one quantization group contains mostly values near zero and a small number with much larger magnitude. If the scale covers the full range, the representable spacing becomes wider for every value in that group. Small values can then collapse onto a limited set of codes.

Clipping the extreme values can make the spacing finer for the majority, but clipped elements acquire larger error. The useful setting depends on the distribution and on how those errors affect attention, not only on reconstruction error averaged across cache elements.

Grouping changes this tension. A smaller group can isolate an outlier from unrelated channels or positions, but it also increases scale metadata. Quantizer design therefore connects statistical structure to a systems cost: finer local ranges require more side information and often more complicated kernels.

Recent-token exceptions change the memory calculation

Some implementations keep a recent portion of the cache at higher precision and quantize only older entries. This creates a residual or staging region.

Such a design can avoid repeatedly quantizing the newest token one element at a time. It can also preserve full precision for positions that have not yet been moved into a packed quantized block. The exact motivation and mechanics depend on the runtime.

A residual region means total memory is not simply cache_elements * low_bit_width. A more representative model is

total =
    high_precision_recent_cache
    + packed_quantized_cache
    + scales_and_metadata
    + temporary_workspace

For short prompts, the fixed or high-precision portion can represent a noticeable fraction of the cache. The asymptotic saving becomes more visible as the quantized portion grows.

Quantization can move work onto the decode path

The KV cache exists to avoid recomputing prefix projections, so access to it sits on a latency-sensitive inference path. A quantized cache reduces bytes read from memory, but attention must still interpret the compressed representation.

One implementation can dequantize cached blocks into a temporary floating-point tensor before attention. Another can fuse unpacking or scale application into an attention kernel. These choices have different workspace, memory-traffic, and kernel-launch behavior.

As a result, a smaller cache does not guarantee lower token latency. Reduced memory traffic can help some workloads, while conversion or packing overhead can dominate others. Hardware, context length, batch shape, cache format, and kernel support determine the actual balance.

The relevant comparison is therefore end-to-end decode behavior under the intended serving shape, not only the byte count of a serialized cache.

Quantized cache state must remain position aligned

Cache entries are indexed by sequence position and layer. Sliding windows, prefix reuse, beam operations, request batching, and cache eviction can all change which logical token occupies a physical cache slot.

Quantization adds scales and packing metadata that must move with the data they describe. Reordering integer codes without applying the same reordering to per-group scales can produce numerically valid tensors associated with the wrong ranges. Similar errors can occur when a packed block crosses a cache boundary assumed by the quantizer.

This is an implementation concern rather than a property of quantization mathematics. The cache manager and quantizer need a shared definition of grouping, physical layout, and token ownership.

Evaluation needs both model output and serving metrics

A cache quantizer can have low reconstruction error yet still alter model outputs in cases where attention is sensitive to a small logit change. Conversely, measurable tensor error does not establish that application-level output has degraded.

Evaluation should therefore cover the model behavior that matters for the application as well as the systems constraint that motivated quantization. Output quality, peak cache memory, decode latency, throughput, and supported context length answer different questions.

The baseline must keep the same model weights, prompts, decoding settings, cache policy, and serving shape except for the cache representation. Otherwise changes from sampling or request scheduling can obscure the effect being measured.

KV cache quantization is most compelling when cached activations, rather than model weights or another workspace, control inference capacity. Once cache storage stops being the dominant allocation, reducing its precision further can add numerical and kernel complexity without addressing the actual memory limit.

Artificial Intelligence Transformers Large Language Models Model Inference