Grouped-Query Attention Shares Key-Value Heads Across Query Groups

Multi-head attention does not require every query head to own a distinct key head and value head. Grouped-query attention, usually abbreviated GQA, partitions query heads into groups and assigns one key-value head to each group. Query projections remain separate, but several query heads read from the same projected keys and values.

That distinction changes parameter shapes and cached state without collapsing the query heads into one attention computation. Each query head still produces its own attention scores because its query vector is different.

Head counts define the sharing pattern

Let a layer have Hq query heads and Hkv key-value heads, with Hq divisible by Hkv. A common grouping assigns:

query heads per KV head = Hq / Hkv

For Hq = 8 and Hkv = 2, one possible mapping is:

Q0 Q1 Q2 Q3 -> K0, V0
Q4 Q5 Q6 Q7 -> K1, V1

The exact tensor layout is implementation-specific, but the logical relationship is that four query heads consume the same key and value projections in this example.

Standard multi-head attention is the endpoint where Hkv = Hq: every query head has its own key-value head. Multi-query attention is the other endpoint where Hkv = 1: all query heads share one key-value head. GQA occupies the intermediate case.

These names describe head sharing, not a complete transformer architecture. Positional encoding, attention masking, head dimension, output projection, and other layer choices remain separate design dimensions.

Shared keys do not imply shared attention weights

Two query heads mapped to the same key head receive the same key vectors for a given sequence position, but their query vectors can differ. Their score calculations therefore remain distinct.

For query head h assigned to KV group g(h), a simplified score expression is:

score_h = Q_h K_g(h)^T / sqrt(d)

The softmax is applied to the scores for that query head under the model’s mask. Different Q_h values can produce different score distributions even when K_g(h) is shared. The resulting weighted value aggregation can also differ because the attention weights differ.

This is the central boundary of the mechanism: GQA shares key and value representations across a group, not the complete attention result. Treating every query head in a group as identical would remove distinctions that the separate query projections still encode.

KV cache size follows KV heads

During autoregressive decoding, a serving runtime commonly stores prior key and value vectors so they do not need to be recomputed for every new token. For a fixed number of cached positions, head dimension, element type, and layer count, the key-value portion of this state scales with Hkv, not Hq.

A conceptual per-layer element count for T cached positions is:

KV elements = 2 * T * Hkv * d

The factor 2 accounts for keys and values. Real allocators add layout, block, alignment, metadata, and batching details, so this expression describes tensor state rather than total process memory.

Reducing Hkv while holding the other terms fixed reduces the logical KV tensor size. That is one reason GQA matters in model serving: query-head count can remain larger than the number of cached key-value heads.

The reduction does not establish an unconditional throughput ratio. Kernel implementation, memory bandwidth, batching, cache layout, sequence lengths, hardware, and scheduler behavior all affect observed serving performance.

Projection parameters change with the KV count

Head sharing also changes the output width of the key and value projections. If each head has dimension d, the query projection produces Hq * d features, while each of the key and value projections produces Hkv * d features.

Conceptually:

Q width = Hq  * d
K width = Hkv * d
V width = Hkv * d

A model implementation may fuse these projections into one matrix or arrange dimensions differently in memory. The logical widths still describe the amount of per-token query, key, and value state represented by the attention configuration.

This means GQA is not merely a cache compression switch applied after a model has been defined. The number of KV heads is part of the layer’s parameterization. Converting an arbitrary existing checkpoint to a different KV-head count requires a defined weight transformation or a model specifically constructed for that configuration; changing a runtime integer alone does not preserve the original computation.

Broadcasting can be logical rather than physical

Attention kernels need to pair each query head with its assigned key-value head. A simple implementation can conceptually repeat KV tensors across the query-head dimension, but materializing repeated copies would give back part of the memory traffic that sharing is intended to avoid.

Implementations can instead express the mapping through indexing, views, strides, or kernels that understand grouped heads. In that case, several query heads consume one KV representation without creating a full physical copy for each head.

This distinction matters when reading tensor traces. A repeated logical shape does not necessarily imply repeated storage, and an API operation named repeat or expand does not by itself establish the allocation behavior without its implementation semantics.

Head sharing is separate from token sharing

GQA changes sharing across attention heads at the same sequence positions. It does not reduce the number of token positions that a head is allowed to access. A full-context GQA layer can still retain KV state for every active prior position, subject to the model and runtime’s context rules.

Conversely, a local-attention layer can use GQA while also restricting each query to a bounded token window. The two mechanisms act on different axes: GQA reduces the number of distinct KV heads, while a local window can reduce the number of live positions needed by a layer.

This separation prevents a common accounting error. KV memory depends on both head structure and position count. Reducing one dimension does not imply that the other dimension is bounded.

The serving boundary is the model’s actual head topology

A serving engine can optimize GQA only if it preserves the checkpoint’s mapping between query heads and KV heads. Cache allocation, attention kernels, tensor-parallel partitioning, and model metadata all need a consistent interpretation of Hq, Hkv, and head dimension.

For capacity planning, the useful quantity is therefore not the advertised number of attention heads alone. The distinct KV-head count determines how many key and value vectors exist per cached position, while the query-head count determines how many query projections and attention outputs remain active.

GQA makes those counts intentionally different. Its practical effect comes from that asymmetry: query diversity can remain represented by multiple query heads while key-value state is shared at a coarser granularity.