Autoregressive transformer inference retains key and value tensors from earlier tokens so each new token can attend to prior context without recomputing those projections. As context length and concurrent sequence count rise, this KV cache can become a substantial part of accelerator memory.
KV cache quantization stores those tensors at reduced precision and reconstructs approximations when attention consumes them. The memory arithmetic is attractive, but the resulting error is not a generic model-weight perturbation. Quantized keys affect attention scores before the softmax, while quantized values affect the weighted sum after attention probabilities have been formed.
That distinction gives developers a more useful way to reason about cache precision than treating every cached tensor as equally sensitive.
Cache size scales with stored elements and bit width
For a decoder with L transformer blocks, assume each block stores H_kv key-value heads, each head has dimension D, and a sequence currently contains T cached tokens. Ignoring metadata and allocator overhead, the cache stores this many scalar elements:
2 * L * T * H_kv * DThe factor of two represents keys and values. Multiplying by bytes per stored element gives the payload size for one sequence.
Reducing a cache from a 16-bit format to an 8-bit representation roughly halves the payload occupied by those elements. Moving to 4-bit storage roughly quarters it. Actual allocated memory can differ because low-bit formats may require scales, zero points, padding, packing, alignment, or temporary dequantization buffers.
The useful comparison is therefore not only nominal bit width. It is total bytes retained per cached token under the concrete representation used by the inference engine.
Key error changes the softmax input
For one attention head, a query vector q scores a cached key k_i through a scaled dot product:
s_i = (q dot k_i) / sqrt(D)If quantization reconstructs the key as k_i + e_i, the score becomes:
s_i' = s_i + (q dot e_i) / sqrt(D)The effect of key error therefore depends on its projection onto the current query. A key reconstruction error with a large norm can have little score effect for one query if the error is nearly orthogonal to that query. A smaller error aligned with the query can change the score more.
Those perturbed scores then pass through softmax. Softmax couples positions: changing one score can alter the normalized probability assigned to several cached tokens. The practical error budget for keys is consequently tied to attention-score distortion, not just elementwise reconstruction error.
This also means a cache format evaluated only with tensor mean squared error can miss behavior that matters to attention. Reconstruction metrics remain useful diagnostics, but they do not directly express the downstream score perturbation for the queries a model actually produces.
Value error enters after attention probabilities
Values occupy a different position in the computation. With attention probabilities p_i, the head output is:
output = sum_i p_i * v_iIf a cached value is reconstructed as v_i + r_i while the probabilities remain fixed, its direct contribution to output error is:
sum_i p_i * r_iA value attached to a position with tiny attention probability contributes little direct error for that query. An error at a heavily weighted position contributes more. Errors from several positions can also reinforce or partially cancel in the vector sum.
This separation does not imply that values are universally safe to quantize more aggressively than keys. Sensitivity depends on the model, cache format, calibration data, context, and operating point. It does show that key and value precision need not be treated as one inseparable parameter.
Quantization granularity controls the range each scale must cover
A low-bit integer code represents a limited set of levels. Mapping a tensor into those levels requires a scale, and sometimes a zero point. The group of elements sharing those parameters determines the range that one quantizer must represent.
A single scale over a large tensor is cheap in metadata, but one extreme magnitude can stretch the represented range and leave fewer effective levels for smaller values. Finer groups let scales adapt to local ranges, at the cost of additional metadata and quantization work.
For KV caches, useful grouping choices can follow token, head, channel, or fixed-size element groups, depending on the engine and format. These choices are not interchangeable. A per-token scale adapts as each new token arrives, while a scale shared across many tokens must accommodate values observed at different positions.
The representation also needs a policy for extreme values. Clipping narrows the covered range and increases resolution inside it, but values outside that range saturate. Expanding the range avoids that clipping while making adjacent quantization levels farther apart. The suitable balance depends on the actual cache distribution and the error tolerated by the model.
Dynamic cache growth changes the calibration problem
Model weights are fixed during inference, so weight quantization can derive parameters from a known tensor before requests arrive. KV cache entries are generated online from request-dependent hidden states. Future cache values are not available when a request starts.
That makes static ranges an assumption about future activations. If runtime values exceed those ranges, clipping or saturation can increase. Dynamic scales adapt to observed cache entries but add scale computation and metadata, and their grouping determines how often that work occurs.
Long contexts add another dimension. A quantization scheme that behaves acceptably near the start of a sequence still needs evaluation across the context lengths used by the application. Cache entries persist, so early quantization decisions can remain relevant many decoding iterations later.
Dequantization cost belongs in the latency model
Compressed storage does not guarantee lower end-to-end latency. Attention kernels must consume the cache representation somehow. An implementation might dequantize values into a wider type before arithmetic, fuse unpacking and scaling into an attention kernel, or use hardware operations that support the stored format more directly.
These paths have different memory traffic, arithmetic, temporary-storage, and kernel-launch characteristics. A format that saves cache capacity can still add enough conversion work to hurt latency on a particular device or sequence shape.
The comparison also changes with batch size and context length. At short contexts, cache traffic may be a small part of total decoding work. At longer contexts, reading cached keys and values becomes a larger operation. Claims about speed therefore need measurements at the sequence lengths, concurrency levels, kernels, and hardware that match the deployment target.
Evaluation should separate capacity gains from model effects
A cache experiment is easier to interpret when it reports two classes of outcome separately. The first is systems behavior: retained bytes per token, maximum feasible context or concurrency, decode latency, and any temporary memory introduced by conversion. The second is model behavior under the same prompts and decoding configuration.
Comparisons also need the same cache policy throughout a run. Mixing precisions across layers, retaining a recent window at higher precision, or exempting selected tensors can be valid designs, but each changes the error path and memory equation. Those policies should be treated as part of the representation rather than hidden implementation details.
KV cache quantization is most useful when its precision is expressed as an explicit resource and error budget. The stored bit width determines only the first part. Grouping, range selection, key-score distortion, value reconstruction, conversion kernels, and context length determine what that bit width means for an actual decoder.