https://nalar.dev/tags/gqa/ Recent content in GQA on Nalar Hugo en-us Wed, 23 Sep 2026 00:00:00 +0000 https://nalar.dev/grouped-query-attention-shares-kv-heads-across-query-groups/ Wed, 23 Sep 2026 00:00:00 +0000 https://nalar.dev/grouped-query-attention-shares-kv-heads-across-query-groups/ <p>Grouped-query attention (GQA) changes a specific structural ratio inside an attention layer: the number of query heads can exceed the number of key and value heads. Several query heads then consume the same projected key and value head. The attention calculation remains head-specific on the query side, while KV state is shared within each group.</p> <p>That asymmetry matters during autoregressive decoding because cached keys and values persist for prior tokens. Reducing the count of distinct KV heads reduces the amount of per-token KV state that must remain available to later decoding steps.</p>