Multi-Head Latent Attention changes the state retained across autoregressive decoding. Instead of requiring a full key tensor and value tensor for every cached token and attention head, the architecture can retain a lower-dimensional latent representation and derive head-specific key/value information through projection structure.
That distinction matters at the cache boundary. Ordinary multi-head attention commonly stores already projected per-head keys and values. MLA moves part of that representation behind a compact bottleneck, so persistent state and compute no longer have the same shape.
The cache boundary moves into a latent space
For a hidden state h_t at token position t, a simplified conventional path can be written as:
K_t = h_t W_K
V_t = h_t W_V
cache <- (K_t, V_t)The cache therefore grows with the dimensions of the projected key and value state.
An MLA-style path introduces a compressed state:
c_t = h_t W_D
cache <- c_twhere W_D is a down-projection. Head-specific transformations can then recover the representations required by attention from c_t through additional projection structure.
This is an architectural compression boundary, not a generic storage codec. The latent vector is part of the model’s trained computation. Replacing an arbitrary MHA cache with a smaller tensor does not reproduce MLA unless the checkpoint and execution graph were constructed for that representation.
Low-rank structure trades retained bytes for projection work
A full KV cache stores information after expansion into the dimensions consumed by attention heads. MLA factors part of those projections through a smaller intermediate dimension.
Conceptually, a matrix can be represented through two factors:
W ~= W_D W_Uwith an intermediate rank smaller than the original projection width. In MLA, the concrete factorization and tensor layout are model-specific, and equality or approximation properties depend on the architecture and training process.
The serving consequence is direct: fewer persistent elements can be retained per token, while some projection work is shifted into the execution path. This changes the balance between memory traffic and arithmetic. It does not imply a fixed latency improvement on every device. Kernel fusion, batch shape, context length, tensor parallel layout, memory bandwidth, and accelerator throughput all affect the measured result.
Position information complicates key compression
Rotary position transforms are not arbitrary linear operations that can always be moved through every projection without changing the function. If a key representation is compressed before a position-dependent transform, reconstructing or transforming it later must preserve the model’s positional semantics.
MLA designs therefore need an explicit boundary between content represented in the compressed latent state and key components that carry positional information. The exact split belongs to the architecture. A runtime cannot freely relocate RoPE, merge its dimensions into another tensor, or cache a different intermediate merely because the resulting shapes are convenient.
This also separates MLA from ordinary cache quantization. Quantization changes the numeric representation of cached values and introduces reconstruction error according to its format. MLA changes which model state is cached in the first place.
Head diversity remains after compression
A shared latent state does not mean all attention heads become identical. Head-specific projections can map the same compact state into different key or value subspaces.
For latent state c_t, a schematic expansion is:
K_t^(i) = c_t U_K^(i)
V_t^(i) = c_t U_V^(i)for head i. Distinct projection parameters preserve head-specific transformations even though the persistent source state is shared.
This differs from Grouped-Query Attention. GQA reduces the number of distinct KV heads and lets multiple query heads consume the same KV head. MLA instead uses a compressed latent representation plus projection structure. Both can reduce KV-cache pressure, but their parameterization and cache semantics are different.
Decode execution can absorb projections in different places
Matrix multiplication associativity can permit some linear operations to be rearranged without explicitly materializing every expanded tensor. For example, an implementation may combine compatible projection matrices ahead of execution or move a projection to the query/output side when the algebra and model graph permit it.
Such rearrangements are implementation choices constrained by mathematical equivalence. Nonlinear operations, positional transforms, quantization boundaries, precision changes, and tensor-parallel communication can block otherwise tempting rewrites.
As a result, the phrase “reconstruct the full KV tensor” is not a universal runtime requirement. Some implementations may avoid materializing that representation. The architectural invariant is that the resulting computation matches the checkpoint-defined attention function within the runtime’s stated numeric behavior.
Cache size and bandwidth are separate from total model memory
Compressing KV state reduces one memory component that grows with active sequence length and concurrent requests. It does not shrink model weights, temporary activations, allocator metadata, workspace buffers, or every communication buffer.
The benefit therefore depends on the serving regime. Long contexts and many simultaneous decode sequences can make KV state a major capacity and bandwidth constraint. Short contexts or compute-heavy execution can expose other bottlenecks first.
A correct comparison must also use the actual latent dimension and any additional cached positional state. Counting only the compact content vector while omitting other persistent tensors understates the cache footprint.
MLA’s technical boundary is specific: it changes attention parameterization so persistent key/value information can pass through a compact latent state before head-specific use. The smaller cache is a consequence of that architecture, while runtime speed remains a property of the complete execution strategy and hardware.