Grouped-query attention (GQA) changes a specific structural ratio inside an attention layer: the number of query heads can exceed the number of key and value heads. Several query heads then consume the same projected key and value head. The attention calculation remains head-specific on the query side, while KV state is shared within each group.
That asymmetry matters during autoregressive decoding because cached keys and values persist for prior tokens. Reducing the count of distinct KV heads reduces the amount of per-token KV state that must remain available to later decoding steps.
Head counts define the grouping boundary
Let an attention layer have (H_q) query heads and (H_{kv}) key-value heads, with (H_q > H_{kv}). In a regular grouping where (H_q) is divisible by (H_{kv}), each KV head serves
[ G = \frac{H_q}{H_{kv}} ]
query heads.
For query head (h), an implementation can associate it with KV head
[ g(h)=\left\lfloor \frac{h}{G}\right\rfloor. ]
The attention operation for that query head then uses its own query vector with the key and value vectors from group (g(h)):
[ A_h=\operatorname{softmax}\left(\frac{Q_h K_{g(h)}^\top}{\sqrt{d}}\right), \qquad O_h=A_hV_{g(h)}. ]
This is an architectural sharing rule, not merely a storage optimization. The model produces fewer distinct key and value projections than a multi-head attention layer with one KV head per query head.
GQA sits between MHA and MQA
Multi-head attention (MHA) commonly uses the same count for query, key, and value heads. At the other endpoint, multi-query attention (MQA) uses multiple query heads with a single shared key-value head.
GQA occupies the intermediate case. It retains more than one KV head but fewer KV heads than query heads. The 2023 GQA paper by Ainslie and colleagues describes this arrangement as a generalization of MQA with an intermediate number of KV heads.
The endpoints can be written compactly:
[ H_{kv}=H_q \quad \text{for MHA}, ]
[ 1 < H_{kv} < H_q \quad \text{for GQA}, ]
[ H_{kv}=1 \quad \text{for MQA}. ]
These equations describe head sharing only. They do not establish identical model quality, latency, or memory behavior across implementations.
KV cache size follows the KV-head count
During cached autoregressive decoding, each layer retains key and value vectors for previous token positions. Ignoring metadata, alignment, quantization details, and implementation-specific padding, the stored scalar count for one layer scales as
[ 2 \times T \times H_{kv} \times d, ]
where (T) is the cached sequence length and (d) is the dimension of each KV head. The factor of two accounts for keys and values.
If two architectures keep (T), (d), element format, and layer count fixed, reducing (H_{kv}) reduces the persistent KV scalar count proportionally. This is the direct memory effect of KV-head sharing.
The relation is narrower than a claim about total serving memory. Runtime memory also includes model weights, temporary attention buffers, allocator overhead, request metadata, and other state. Some kernels may materialize repeated or expanded views during computation even when persistent storage keeps only the compact KV representation.
Sharing does not make query heads identical
Query heads in the same group can still use different query projections. As a result, they can produce different attention logits against the shared keys and different probability distributions over token positions.
For two query heads (a) and (b) mapped to the same KV group (j),
[ Q_a \neq Q_b ]
can yield
[ Q_aK_j^\top \neq Q_bK_j^\top. ]
The shared KV projection constrains the key-value representation available to those query heads, but it does not collapse their query vectors into one head. Their outputs can therefore differ even though both read from (K_j) and (V_j).
This distinction is important when inspecting tensor shapes. Expanding a compact KV tensor across query groups for a kernel interface does not create new independently projected KV heads. It can be a view or repetition of the same group data.
Projection parameters also change
KV-head sharing affects more than the decode cache. A projection layout that emits fewer key and value heads requires fewer output channels for the K and V projections, assuming the per-head dimension stays fixed.
If the model width feeding the projections is (D), separate dense K and V projections have output width (H_{kv}d). Their parameter count therefore scales with (H_{kv}). The Q projection can retain width (H_qd).
Exact parameter layouts vary. Some implementations combine Q, K, and V into one packed matrix; others store separate matrices or use tensor-parallel sharding. The architectural invariant is the number of distinct projected KV heads, not a particular checkpoint tensor name or packing order.
Tensor parallelism adds divisibility constraints
Serving systems often shard attention heads across devices. A GQA configuration that is mathematically valid for a single device can require extra handling when (H_{kv}) does not divide cleanly across the chosen tensor-parallel degree.
A runtime may replicate KV heads across ranks, choose a different partition, or reject unsupported head-count combinations. Those choices belong to the runtime contract. GQA itself specifies sharing between query groups and KV heads; it does not prescribe a distributed sharding policy.
The same separation applies to kernel behavior. A kernel may accept compact GQA tensors directly, or a framework may expand KV references to match a more uniform attention interface. Persistent cache size and transient execution layout are therefore separate properties.
Cache traffic remains an implementation boundary
Fewer cached KV heads reduce the distinct KV data associated with each token position. That can reduce bytes that need to be stored and can reduce data movement in decode paths designed to exploit the compact representation. It does not guarantee a fixed latency ratio.
Decode latency also depends on batch shape, sequence length, kernel fusion, memory hierarchy, quantization, scheduling, device utilization, and the attention implementation. A runtime that expands shared KV data before the main kernel can expose different traffic characteristics from one that consumes grouped heads natively.
GQA’s stable technical boundary is structural: several query heads share each key-value head. That lowers the number of distinct KV projections and the persistent KV state relative to otherwise comparable MHA, while preserving multiple KV groups rather than collapsing the layer to the single KV head used by MQA.