Autoregressive decoding retains key and value tensors from earlier tokens, so KV-cache memory grows with retained context. Quantizing those tensors changes a direct term in that memory footprint: fewer bits are stored for each cached element. The trade is not free capacity. Reduced precision adds representation error and requires a concrete scaling, storage, and kernel strategy.
Cache precision is separate from weight precision
Model weights and KV state have different lifetimes. Weights are persistent across requests, while KV tensors are generated from each request and grow as its sequence advances. A model can therefore use one numerical format for weights and another for its cache.
For a simplified cache with L layers, H_kv KV heads, head dimension D, retained length T, and b bytes per stored element, the payload is proportional to:
KV_bytes = 2 * L * H_kv * D * T * bThe factor 2 accounts for keys and values. Changing b from two bytes to one halves this tensor payload under the same shape. That arithmetic does not include scale metadata, alignment, allocator granularity, or runtime workspaces.
Quantization needs a scale convention
Mapping a floating-point tensor into a smaller numerical domain requires parameters that relate stored values to the represented range. A common affine-free form can be written conceptually as:
q = clamp(round(x / s))
x_hat = q * swhere s is a scale, q is the stored quantized value, and x_hat is the reconstructed approximation.
The scale can be attached at different granularities: per tensor, per head, per channel, per token group, or another implementation-defined partition. Finer granularity can track local ranges more closely but requires more metadata and potentially more indexing or conversion work.
This choice is part of the quantization scheme, not a property of attention itself.
Keys and values need not have identical error sensitivity
Keys participate in query-key dot products that produce attention logits. Values are combined after attention weights have been formed. Quantization error therefore enters the computation through different paths.
For a query q_t, reconstructed cached key k_hat_i, and reconstructed value v_hat_i:
score_i = q_t^T k_hat_i / sqrt(D)
output = sum_i softmax(score)_i * v_hat_iError in a key can perturb the logits and thus the normalized weight distribution. Error in a value perturbs the vectors mixed by those weights. A cache format or scaling policy can treat K and V differently when an implementation has evidence that doing so is beneficial.
No fixed bit width guarantees equivalent model quality across architectures, context lengths, tasks, or calibration choices.
Lower payload does not imply proportional latency reduction
A smaller cache can reduce memory traffic during decoding because attention reads cached K and V repeatedly. That creates an opportunity for higher effective bandwidth or greater concurrency.
The realized latency effect also depends on the kernel. If cached values must be converted to a wider format before arithmetic, conversion instructions, scale loads, unpacking, and layout handling consume work. Hardware support for the stored format can materially change this balance.
A twofold reduction in cache payload therefore does not establish a twofold speedup. Memory capacity, memory bandwidth, arithmetic throughput, kernel fusion, batch shape, and context length all affect the result.
Metadata and block layout remain part of physical memory use
Quantized elements are only one component of the cache allocation. Scales consume storage, and some formats carry additional metadata. Block allocators can round allocations to fixed granularities, while alignment constraints can leave padding.
Consequently, theoretical tensor bytes and observed device-memory savings are different quantities. The first follows from element count and representation width. The second is an implementation measurement that includes the allocator and cache format.
Paged or block-based cache management is compatible with quantized storage, but paging and quantization solve different problems. Paging changes placement and allocation granularity. Quantization changes the representation of the KV elements stored inside those allocations.
Quantization error accumulates through model computation, not by a simple byte ratio
Reducing cache precision changes the numerical state consumed by later decoding steps. The effect cannot be inferred from memory savings alone. Error can interact with attention logits, residual pathways, later layers, and subsequent token choices.
Evaluation therefore needs to keep two claims separate. Memory reduction can often be computed directly from the chosen format plus metadata. Output quality must be measured for the model, cache scheme, context regime, and workload being served.
KV-cache quantization has a precise systems boundary: it trades representation fidelity and conversion work for fewer cache bytes per retained token. Its capacity effect is structural; its quality and latency effects remain properties of the selected quantizer, kernels, hardware, and model.