Autoregressive decoding appends key and value tensors to a cache at every transformer layer. The cache prevents prior tokens from being projected into keys and values again, but its storage grows with sequence length. KV cache quantization changes that storage representation: older or selected cache entries are encoded with fewer bits, then reconstructed when attention consumes them.
The mechanism is a memory-format trade. It does not remove tokens from the attention context and it does not change the model weights. It reduces bytes used by cached state while introducing quantization error, scale or zero-point metadata, and conversion work on the decode path.
Cache growth makes precision a systems parameter
For a decoder with batch size (B), sequence length (T), (L) cached layers, (H_{kv}) key-value heads, and head dimension (D), a simple dense cache stores both keys and values:
[ N_{KV} = 2 B T L H_{kv} D ]
elements.
If each element uses (p) bytes, the payload is approximately:
[ M_{KV} = 2 B T L H_{kv} D p ]
before allocator effects and auxiliary metadata. The linear factor in (T) makes cache representation increasingly important as contexts, batches, or concurrent requests grow.
Changing a cached element from a 16-bit representation to a low-bit code reduces the payload term, but the ratio is not exactly the ratio of bit widths in a real implementation. Quantization groups also need scales and, for affine schemes, zero points. Padding, packing granularity, temporary reconstruction buffers, and a full-precision residual region can add more storage.
Quantization maps groups into a smaller code space
A common affine scheme assigns a scale (s) and zero point (z) to a group of cache values. A stored integer code can be written conceptually as:
[ q = \operatorname{clamp}(\operatorname{round}(x/s) + z) ]
and reconstruction as:
[ \hat{x} = s(q-z) ]
The reconstructed (\hat{x}) generally differs from the original (x). Attention therefore operates on an approximation of the cached keys or values after reconstruction.
Grouping controls the range shared by one set of quantization parameters. A larger group amortizes metadata across more elements but forces more values to share a scale. A smaller group can adapt to local ranges more closely while increasing metadata and quantization work.
This is not merely a file-format decision. The grouping axis interacts with the statistical structure of key and value tensors. KIVI, for example, reports a per-channel treatment for keys and a per-token treatment for values based on observed outlier patterns in the evaluated models. That result is a property of the method and its measurements, not a universal requirement for every cache quantizer.
Key error perturbs attention scores
For one attention head, scores depend on the query (Q) and cached keys (K):
[ S = QK^\top / \sqrt{d} ]
With reconstructed keys (\hat{K} = K + E_K), the score becomes:
[ \hat{S} = Q(K + E_K)^\top / \sqrt{d} ]
so the score perturbation contains:
[ \Delta S = QE_K^\top / \sqrt{d} ]
before softmax.
The softmax then converts score changes into changes in attention weights. The effect is data-dependent: equal quantization error magnitudes do not imply equal output changes because queries, score margins, and the distribution of competing keys differ.
This boundary matters when describing cache compression. A low average tensor reconstruction error is not itself a guarantee of identical token probabilities or identical generated sequences.
Value error enters after attention weights
Values occupy a different position in the attention computation. If (P) denotes the attention-weight matrix, the output is:
[ O = PV ]
and reconstructed values (\hat{V} = V + E_V) give:
[ \hat{O} = P(V + E_V) = O + PE_V ]
when holding (P) fixed.
Key error can alter the weights through the score path, while value error is mixed by the weights after softmax. A concrete quantizer may therefore use different grouping or precision policies for keys and values. Treating both tensors identically is an implementation choice, not an architectural rule of transformer attention.
A residual cache moves conversion off the newest tokens
Some runtime designs keep a bounded region of recent keys and values in the compute precision and quantize older entries in batches. Hugging Face Transformers exposes this pattern through a residual cache in its quantized-cache implementation.
Conceptually, the state is split into two regions:
older tokens recent tokens
+----------------------+ +------------------+
| low-bit K,V + params | | original K,V |
+----------------------+ +------------------+
| |
+------ attention --------+This arrangement changes both memory use and execution cost. A larger residual region retains more full-precision state and consumes more memory. A smaller region moves more state into the compact representation and can trigger conversion more frequently, depending on the implementation.
The residual region is not inherent to KV cache quantization. It is one runtime strategy for balancing conversion overhead, recent-state precision, and memory pressure.
Lower storage does not guarantee lower decode latency
Quantization reduces bytes that must remain resident for the compact portion of the cache. That can be valuable when cache capacity is the limiting resource. The decode kernel, however, still needs values in a form usable by its arithmetic path.
A runtime may dequantize into a compute type, fuse reconstruction with attention, or use another specialized kernel strategy. Each choice changes memory traffic, temporary storage, launch overhead, and arithmetic cost. For short contexts that already fit comfortably in accelerator memory, conversion overhead can outweigh the benefit of moving fewer cache bytes. Current Transformers documentation explicitly notes that quantized cache can hurt latency in that regime.
For long contexts or high concurrency, reduced cache footprint can instead increase feasible batch size or prevent allocation failure. Those are deployment effects, not automatic consequences of selecting a nominal bit width.
Bit width alone does not define the cache format
Two caches described as 4-bit can differ materially. Relevant parameters include group size, grouping axis, symmetric or affine mapping, metadata precision, residual-cache length, packing layout, reconstruction type, and kernel support.
Model architecture also changes the baseline. Multi-query and grouped-query attention reduce the number of stored KV heads before quantization is applied. Sliding-window layers can bound the number of retained positions. Quantization composes with those mechanisms, but it does not replace them.
The practical memory figure is therefore the compact payload plus its metadata and any full-precision or temporary regions. The practical latency figure includes the attention kernel and all representation conversions on its path.
KV cache quantization is most accurately treated as a storage representation for attention state. Its gain is reduced resident state; its cost is approximation plus representation-management work. The balance depends on cache geometry, quantizer granularity, kernel implementation, context length, and available memory rather than on bit width in isolation.