A batch can contain two prompts with very different token counts yet represent both with the same rectangular tensor. The shorter prompt is extended with padding so its tensor shape matches the longest sequence in the batch. An attention mask can stop those padded positions from contributing to attention probabilities, but that semantic exclusion does not imply that dense kernels skip every operation associated with the padded rows and columns.
This distinction matters in model serving because the mask controls which positions may interact, while the tensor shape controls much of the work presented to dense operators. A batch with substantial padding can therefore execute more arithmetic and move more data than its count of real tokens suggests.
Masking changes attention eligibility
For one attention head, a common formulation starts from query, key, and value matrices:
Q = XWq
K = XWk
V = XWv
A = softmax(QK^T / sqrt(d) + M)
O = AVM is a mask term. Entries representing disallowed query-key pairs receive a value that makes their probability effectively zero after softmax. The exact representation varies by implementation: some paths use a large negative finite value, some use negative infinity, and fused kernels can encode the same rule without materializing a full mask matrix.
The semantic result is narrow. A masked key position does not contribute probability mass to an allowed query after the masking rule is applied correctly. That says nothing by itself about whether projections, score tiles, normalization state, or output rows for padded positions were computed.
A mask is therefore not a general compute-pruning contract. It specifies valid attention relationships.
Rectangular batches still expose padded dimensions
Suppose a batch contains sequences with lengths 128 and 512. A conventional padded representation can have shape:
batch x 512 x hidden_dimensionThe first sequence has 384 padded positions. If the implementation applies dense projection operations to the full tensor, Q, K, and V are produced for those positions as part of the same matrix operations used for real tokens.
Attention kernels also receive dimensions derived from the padded sequence length unless the runtime carries separate sequence-length metadata into a kernel that can exploit it. A logical mask may suppress invalid score contributions while the launched tensor program still covers tiles spanning padded regions.
The exact amount of wasted work is implementation-dependent. Kernel fusion, tiling, sequence packing, variable-length APIs, and backend-specific dispatch can change which operations are skipped. The reliable boundary is simpler: a mask alone does not establish that padded positions disappear from execution.
Causal masks and padding masks solve different constraints
Autoregressive decoders often combine two restrictions. A causal mask prevents a token from attending to future positions. A padding mask prevents attention to positions that do not belong to the sequence.
These masks can be merged into one attention rule, but their purposes remain different. Causality defines dependency direction among valid tokens. Padding identifies tensor positions introduced to make shapes compatible.
Neither rule necessarily changes the physical tensor dimensions. A dense causal attention operation can still process a square score domain while excluding the upper triangle semantically. Specialized kernels can exploit causal structure and avoid some work, but that optimization comes from the kernel implementation, not from the abstract presence of a causal mask. The same separation applies to padding.
This is useful when reading an API surface. Parameters named attention_mask, key_padding_mask, or similar names describe semantics first. Performance behavior has to be established from the runtime and kernel path that consumes them.
Packing removes padding from representation
Sequence packing takes a different approach. Instead of extending every sequence to a shared maximum length, valid tokens from several sequences are stored compactly, accompanied by metadata that preserves sequence boundaries.
A variable-length attention kernel can use cumulative sequence offsets or equivalent metadata to ensure tokens from one sequence do not attend to another. The physical representation then contains fewer padded positions, so the kernel has a chance to avoid work that a rectangular padded layout would expose.
Packing is not interchangeable with masking. It changes layout and indexing. The runtime must preserve boundaries, position semantics, causal structure, and any model-specific assumptions tied to token positions. A kernel must explicitly support the packed or variable-length representation.
This also means that replacing a padded batch with a packed batch is an execution change, not just a different mask value. Correctness has to be checked at the boundary where positions, offsets, and attention ranges are reconstructed.
Padding cost depends on batch composition
Padding overhead is driven by the gap between each sequence length and the batch maximum. Grouping requests with similar lengths can reduce that gap even when the model and attention kernel remain unchanged.
For a batch of B sequences padded to length Lmax, the tensor contains:
B * Lmaxtoken positions, while the number of valid positions is:
sum(L_i)The difference is a useful measure of padded positions:
padding_positions = B * Lmax - sum(L_i)It is not a direct latency or FLOP formula. Different operators scale differently with sequence length, kernels process tiles rather than isolated scalar positions, and hardware utilization can improve as shapes become larger. Still, the count exposes batch composition that a real-token counter hides.
For attention specifically, a naive dense score matrix has a sequence-by-sequence dimension. That makes large padding gaps especially relevant during prefill. Fused attention implementations can avoid materializing the score matrix, yet their execution cost still depends on the sequence dimensions and on any variable-length support they provide.
Generation changes the shape of the problem
During autoregressive generation with a KV cache, each decoding step usually adds a small number of new query positions while attending over cached keys and values. Padding behavior can therefore differ from prefill.
Serving systems may batch active sequences with different cache lengths. Some runtimes use paged or block-based cache layouts, sequence metadata, or specialized kernels so cached tokens do not need to form one simple padded rectangle. Other paths can still incur work related to shared shape bounds.
A prefill benchmark cannot establish decode padding behavior, and a decode benchmark cannot establish prefill behavior. The kernel path, cache representation, and batching policy have to be identified separately.
Measure the execution path, not the mask name
A useful serving measurement records real token counts, padded or bucketed lengths, batch composition, and the actual attention backend. Comparing batches with the same number of real tokens but different length distributions can expose sensitivity to padding without attributing every difference to one operator.
Profiler traces add the next boundary. They can show kernel shapes, dispatch choices, and whether a variable-length path is active. Source or backend documentation can then establish which dimensions a specific kernel skips.
The practical implication is precise: masking can make padded positions semantically inert while leaving them physically present in dense execution. Removing their compute requires an execution mechanism that understands valid sequence lengths, such as packing, variable-length kernels, or another backend-specific form of shape-aware skipping. Correct attention probabilities and reduced padding work are related goals, but they are not the same operation.