Grouped-Query Attention Shrinks the KV Cache by Sharing Key-Value Heads

Grouped-query attention changes the number of key and value heads that must be stored during autoregressive decoding. Instead of giving every query head its own key-value pair, several query heads address the same key-value head. The query side can retain many heads while the persistent KV state uses fewer independent projections.

This is an architectural change to attention, not a cache compression codec. The smaller cache follows from producing fewer distinct key and value heads per token.

Head counts separate query capacity from KV state

Let an attention layer have Hq query heads and Hkv key-value heads, with Hkv dividing Hq. A regular grouped arrangement assigns:

queries_per_kv_head = Hq / Hkv

Each query head still computes its own query vector. Query heads in the same group consume the key and value vectors from one shared KV head.

Multi-head attention is the endpoint where:

Hkv = Hq

Every query head has a distinct key-value head. Multi-query attention is the opposite endpoint:

Hkv = 1

All query heads share one key head and one value head. Grouped-query attention occupies the intermediate region:

1 < Hkv < Hq

The grouping therefore exposes a structural parameter that controls how much independent KV state exists in a layer.

KV-cache size scales with the key-value head count

During incremental decoding, cached keys and values from prior tokens are reused by later tokens. For a batch size B, cached sequence length T, Hkv key-value heads, head dimension D, and element width S bytes, the raw storage for one layer is proportional to:

KV_bytes = 2 * B * T * Hkv * D * S

The factor 2 accounts for keys and values. This expression omits allocator padding, metadata, paging structures, quantization metadata, and implementation-specific layout overhead.

If Hq = 32 and Hkv = 8 with the same D and element format, the raw KV tensor contains one quarter as many head elements as a 32-key-value-head configuration:

8 / 32 = 1/4

That ratio applies to the KV-head component of storage. It does not state that total model memory falls by the same fraction because weights, activations, runtime workspaces, and other state remain.

The reduction also affects decode-time memory traffic. A decoder repeatedly accesses cached keys and values while producing new tokens. Fewer distinct KV heads reduce the amount of KV data that must be represented and can reduce bytes moved for that state. Actual latency remains dependent on kernels, hardware, batch shape, sequence length, cache layout, and the rest of the model.

Sharing occurs before cache storage

The distinction between GQA and a cache deduplication scheme matters. The model projects hidden states into Hkv key heads and Hkv value heads. Those are the tensors stored for later attention operations. There are no separate per-query KV heads that a runtime must first compute and then merge merely to obtain the standard GQA cache representation.

Conceptually, a group can be written as:

Q_0, Q_1, Q_2, Q_3 -> K_0, V_0
Q_4, Q_5, Q_6, Q_7 -> K_1, V_1

for a case with four query heads per KV head. The query projections remain distinct. Sharing applies to the key and value side used by those queries.

Some kernels may expand, broadcast, repeat, or index KV heads in different internal forms. Such implementation choices do not change the architectural head relationship. A materialized repeat can erase part of the memory advantage inside an intermediate buffer, while an indexed or broadcast representation can avoid that expansion. The model definition and kernel realization should therefore be treated separately.

Grouping changes attention parameterization

Reducing Hkv changes the key and value projection parameterization as well as cache geometry. With equal head dimension D, a conventional projection produces key and value widths proportional to Hkv * D. Fewer KV heads mean fewer independent key and value channels at that projection boundary.

This is the central trade: query heads no longer each receive a private key-value projection. Heads inside a group must attend through shared key and value representations, even though their query vectors and resulting attention distributions can differ.

The softmax is still evaluated per query head. Sharing K and V does not force two query heads to produce identical attention weights because their query vectors can differ:

scores_i = Q_i K_g^T / sqrt(D)
weights_i = softmax(scores_i)
output_i = weights_i V_g

Here g identifies the KV group for query head i. Two heads mapped to the same g use the same K_g and V_g, but distinct Q_i can produce distinct scores_i and weights_i.

GQA is an intermediate architectural point, not a universal ratio

The GQA formulation introduced by Ainslie and colleagues generalizes multi-query attention by using more than one but fewer key-value heads than query heads. The paper also describes converting multi-head checkpoints through uptraining, including construction of grouped key-value heads from existing heads. Those conversion procedures are a training strategy, not a requirement that every GQA model must originate from a multi-head checkpoint.

Likewise, no single Hq:Hkv ratio defines GQA. A model can choose its head counts as part of its architecture, subject to grouping and implementation constraints. The resulting quality, throughput, and memory behavior cannot be inferred from the ratio alone.

Shazeer’s earlier multi-query attention work identifies repeated loading of large key and value tensors as an important cost in incremental decoding and shares keys and values across all query heads. GQA retains the same direction of reducing distinct KV state while restoring multiple KV heads between the MQA and MHA endpoints.

Cache savings compose with other KV techniques

GQA reduces the number of KV heads. Other mechanisms can target different dimensions of the same cache. Lower-precision KV storage changes S; sliding or bounded attention changes the effective T; cache paging changes allocation and placement; prefix reuse can avoid storing duplicate prefix state across compatible requests.

These mechanisms are not interchangeable. A model with fewer KV heads can still use a long context and therefore accumulate substantial cache state. A quantized cache can still have many KV heads. A paged cache can improve allocation behavior without reducing the logical number of KV elements.

For capacity analysis, separating the dimensions prevents misleading estimates:

KV capacity ~ layers * B * T * Hkv * D * bytes_per_element

Each optimization acts on one or more factors, while runtime overhead can add terms outside this simplified expression.

Grouped-query attention places the KV-head count directly in the model architecture. Its serving effect follows from that boundary: fewer independent key-value projections are generated per token, fewer KV-head elements need persistent storage, and multiple query heads consume each stored KV representation. The precise speed benefit is an implementation result, but the reduction in logical KV-head state is structural.