Autoregressive language models generate one token at a time. To avoid recomputing attention keys and values for every previous token at every step, inference systems usually keep those tensors in a key-value cache, or KV cache. This saves computation, but the cache grows with sequence length and can become a major memory cost when serving long contexts or many requests at once.
One architectural choice has a direct effect on that cost: how many separate key and value heads the attention layer stores. Standard multi-head attention gives every query head its own key and value head. Grouped-query attention (GQA) keeps multiple query heads but lets groups of them share key and value heads.
That change can make the KV cache much smaller without collapsing attention to a single shared key-value head. This article builds the idea from standard multi-head attention, works through the memory arithmetic, and explains the quality, compatibility, and serving trade-offs that matter in practice.
Start with the three roles in attention
An attention head uses three projected representations:
- a query asks what information the current position needs;
- a key describes what each available position can be matched against;
- a value contains the information returned when a position receives attention.
For one head, scaled dot-product attention can be written as:
Attention(Q, K, V) = softmax(Q K^T / sqrt(d_head)) VDuring autoregressive decoding, the model creates a new query for the current token. Keys and values from earlier tokens do not need to be projected again if they are cached. The inference system therefore appends the new token’s keys and values to the KV cache and reuses the previous ones.
The important consequence is that query tensors are temporary for a decoding step, while cached keys and values persist across steps. Reducing the number of key-value heads can therefore reduce persistent decoding memory even when the number of query heads stays unchanged.
Compare MHA, MQA, and GQA
Suppose an attention layer has eight query heads.
With multi-head attention (MHA), each query head has a corresponding key head and value head:
8 query heads
8 key heads
8 value headsWith multi-query attention (MQA), all query heads share one key head and one value head:
8 query heads
1 key head
1 value headGrouped-query attention sits between those designs. For example, eight query heads can share four key-value heads:
8 query heads
4 key heads
4 value headsEach key-value head serves a group of two query heads. The query heads remain distinct, so they can still form different query projections. What is shared is the set of cached keys and values they attend to.
A useful mental model is:
MHA: one KV head per query head
GQA: one KV head per group of query heads
MQA: one KV head for all query headsMQA is therefore the most aggressive sharing case, while GQA provides intermediate choices.
Work through the KV-cache arithmetic
Consider a simplified decoder with:
layers = 32
sequence length = 4096 tokens
query heads = 32
head dimension = 128
bytes per element = 2Assume standard MHA first, so there are also 32 key heads and 32 value heads.
Ignoring allocator overhead and other implementation details, the KV cache for one sequence is approximately:
2 * layers * tokens * kv_heads * head_dim * bytes_per_elementThe leading 2 accounts for both keys and values.
For MHA:
2 * 32 * 4096 * 32 * 128 * 2
= 2,147,483,648 bytes
= 2 GiBNow keep 32 query heads but use eight key-value heads with GQA:
2 * 32 * 4096 * 8 * 128 * 2
= 536,870,912 bytes
= 512 MiBThe KV-cache tensor storage is one quarter as large because the number of key-value heads fell from 32 to 8.
The general ratio, when head dimension and cache representation stay the same, is:
GQA KV cache / MHA KV cache = g / hwhere h is the number of query heads and g is the number of key-value heads. With h = 32 and g = 8, the ratio is 8 / 32 = 1/4.
This is a teaching calculation, not a promise about total process memory. Real serving systems also allocate model weights, activations, temporary workspaces, metadata, memory-management structures, and sometimes padded or paged cache blocks.
How grouped-query attention maps heads
Let the model have h query heads and g key-value heads, with h divisible by g. Each key-value head is then shared by:
heads_per_group = h / gFor eight query heads and four key-value heads:
query heads 0, 1 -> KV head 0
query heads 2, 3 -> KV head 1
query heads 4, 5 -> KV head 2
query heads 6, 7 -> KV head 3Conceptually, an implementation can make each shared key and value available to every query head in its group before computing attention. That does not mean a serving engine must physically duplicate the cached tensors. Efficient kernels can operate on the grouped layout directly or expand views internally as needed.
This distinction matters. If application code explicitly copies key and value tensors until they have one physical copy per query head, some of the memory benefit can disappear in intermediate tensors even though the model architecture is GQA.
Why fewer KV heads help decoding
The first benefit is straightforward: fewer cached key and value vectors must be stored per token.
There is also a bandwidth benefit. During incremental decoding, attention repeatedly reads cached keys and values for previous positions. Smaller KV tensors mean less key-value data needs to be moved from memory for a given sequence, all else being equal. On workloads where KV-cache traffic is an important bottleneck, that can improve decoding efficiency.
However, architecture alone does not determine observed latency. Kernel implementation, batch size, sequence length, hardware, cache layout, quantization, scheduling, and other operations in the model can all affect the result. GQA reduces a specific source of storage and memory traffic; it does not guarantee a fixed end-to-end speedup.
The memory saving also matters for concurrency. If KV cache is a limiting resource, reducing cache bytes per active token can allow a serving system to hold more tokens or requests in the same device memory. Whether that turns into higher throughput depends on the rest of the serving stack.
Understand the quality trade-off
Why not use one key-value head for every model?
Sharing key and value projections removes some representational freedom. In MHA, each query head can interact with keys and values produced by its own learned projections. MQA shares those key and value projections across all query heads. GQA keeps several distinct key-value groups, providing a middle point between the two designs.
The practical trade-off is therefore not simply “more heads are better.” It is:
more KV heads
-> more independent key/value projections
-> larger KV cache
fewer KV heads
-> more sharing
-> smaller KV cacheHow much sharing a model can tolerate without unacceptable quality loss is an empirical property of the model architecture and training process. A configuration that works well for one model should not be assumed to transfer unchanged to another.
GQA is a model property, not a serving switch
A common mistake is to treat grouped-query attention like an inference option that can be enabled on any multi-head model.
The number and shapes of query, key, and value projections are part of the model architecture and learned parameters. A checkpoint trained with 32 independent key heads does not become an equivalent eight-KV-head model merely because an inference server groups those heads at runtime.
Changing an existing MHA checkpoint into GQA requires a defined conversion and training strategy if model quality is to be preserved. Research has explored uptraining existing checkpoints after combining key-value heads, but that is a model-development operation, not a lossless serving transformation.
For application developers choosing a pretrained model, the practical rule is simpler: inspect the model’s architecture rather than assuming its number of attention heads equals its number of key-value heads.
Do not confuse KV-head count with head dimension
KV-cache size depends on both the number of key-value heads and the dimension of each head.
Two models can both use GQA and still have very different cache costs because they differ in layer count, head dimension, context length, cache precision, or other architecture choices. Comparing only num_key_value_heads is therefore incomplete.
For a rough comparison between models, reason from:
cache bytes per token
≈ 2 * layers * kv_heads * head_dim * bytes_per_elementThen multiply by the number of cached tokens. This estimate is useful for understanding architecture-level scaling, but actual memory allocation should be measured in the serving implementation you plan to deploy.
Separate GQA from KV-cache quantization
GQA and cache quantization reduce KV-cache memory in different ways.
GQA stores fewer key and value heads:
same element format * fewer KV elementsKV-cache quantization stores each cached element in a lower-precision representation:
similar KV structure * fewer bits per stored elementA serving stack may support both, so their benefits can be complementary. They also introduce different constraints. GQA is built into the model architecture, while cache quantization is an inference representation choice whose support and numerical behavior depend on the serving implementation.
Keeping those mechanisms separate makes capacity planning clearer. If a model already uses GQA, switching cache precision does not change how many KV heads the model has.
Common implementation mistakes
Assuming query heads and KV heads are equal
Code written only for MHA may reshape keys and values using the query-head count. That can fail outright or silently produce incorrect layouts for GQA models. Treat query-head count and key-value-head count as separate architecture parameters.
Materializing repeated KV tensors too early
It is mathematically convenient to describe GQA as repeating each key-value head across its query group. Physically copying those tensors can create unnecessary memory traffic. Prefer attention implementations that understand grouped heads when your framework and hardware support them.
Comparing cache sizes without matching precision
A 16-bit MHA cache and an 8-bit GQA cache differ in two variables. If the goal is to isolate the architectural effect of GQA, hold cache precision, sequence length, layer count, and head dimension constant.
Expecting identical output after changing the architecture
Collapsing several trained KV heads into fewer heads changes the computation unless the model was specifically trained or adapted for that structure. Do not treat arbitrary head averaging as a lossless optimization.
When GQA is especially useful
GQA is attractive when autoregressive decoding memory or bandwidth is important: long-context generation, high-concurrency serving, or deployments where KV cache consumes a meaningful share of accelerator memory.
It matters less when KV cache is not the limiting resource. Short prompts with tiny batches, workloads dominated by other model operations, or systems constrained mainly by model-weight memory may see less practical benefit from reducing KV heads.
For developers selecting among pretrained models, GQA is one architecture property to include in inference planning alongside parameter count, context length, numerical precision, and serving-engine support. For model designers, the number of KV heads is a capacity-versus-efficiency choice that should be validated with task quality and realistic inference benchmarks.
Conclusion
Grouped-query attention reduces KV-cache cost by separating the number of query heads from the number of key-value heads. Multiple query heads share each cached key-value head, so the model can retain several query projections while storing fewer key and value vectors per token.
The key mental model is simple: KV-cache size scales with KV heads, not query heads alone. That makes GQA valuable for memory-conscious LLM inference, but it is an architectural trade-off rather than a free runtime toggle. Measure it in the context of the actual model, cache representation, serving engine, and workload.