A dot product between a query and a key tends to grow in magnitude as their dimension grows. In scaled dot-product attention, the score is divided by the square root of the key dimension before softmax. That factor is not a cosmetic normalization. It controls the scale presented to softmax under a specific statistical assumption about the query and key components.

The familiar expression is

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

where d_k is the query-key dimension for one attention head. The scaling term affects the distribution of attention probabilities even though it does not change the ordering of logits by itself.

Dot-product variance grows with dimension under a simple model

Consider one query q and key k, each with d_k components. Their unscaled score is

s = q · k = sum_i q_i k_i

Suppose, for analysis, that the component products are independent, have mean zero, and each has variance near one. Then the variance of their sum is approximately

Var(s) = d_k

and the standard deviation grows like sqrt(d_k).

Those assumptions are an analytical model, not a guarantee about activations in a trained transformer. Real query and key coordinates can be correlated, have non-unit variance, and change distribution across layers, tokens, checkpoints, or quantized implementations. The calculation still explains the role of the conventional scaling factor: dividing by sqrt(d_k) removes the direct dimension-dependent growth predicted by that simple model.

With

s_scaled = s / sqrt(d_k)

the same assumptions give variance near one instead of variance proportional to d_k.

Softmax reacts to scale, not only ordering

Multiplying every logit by the same positive constant preserves their rank, but softmax probabilities change. For logits z_i,

p_i = exp(z_i) / sum_j exp(z_j)

Larger separations between logits push more probability mass toward the largest entries. Smaller separations produce a flatter distribution.

This makes score scale part of attention behavior. If query-key dot products become wider merely because d_k increased, softmax can become more concentrated even when the directional relationships among vectors have not changed in a corresponding way.

The 1 / sqrt(d_k) factor compensates for that dimensional effect under the assumptions above. It does not force a fixed entropy, guarantee nonsaturated probabilities, or make attention distributions identical across head sizes. The actual logits still depend on model parameters and activations.

The factor is tied to head dimension

In multi-head attention, d_k refers to the width of a query or key within a head, not necessarily the model’s full hidden size. If a hidden representation of width d_model is split into heads, each head computes its own query-key scores in its projected dimension.

Using sqrt(d_model) in place of sqrt(d_k) changes the effective softmax temperature whenever those values differ. The result may still execute correctly at the tensor level, but it is no longer the same scoring rule.

This distinction also matters in architectures where query heads and key-value heads are arranged differently. Grouped-query or multi-query attention can share keys and values across query heads, yet the scale still follows the dimension of the vectors participating in each query-key dot product. Head count and key-sharing topology do not by themselves define the denominator.

A softmax with temperature T can be written as

softmax(z / T)

so the standard attention denominator acts like a dimension-derived temperature applied to raw dot products. Adding another temperature term changes the effective scale again:

softmax((q · k) / (sqrt(d_k) T))

For positive T, values above one flatten the distribution relative to the standard scale, while values below one sharpen it. This can be useful as an explicit model or serving choice, but it should not be confused with merely preserving the conventional attention definition.

A trainable or architecture-specific scale can also replace or modify the standard factor. In that case, behavior follows that model’s definition. The square-root rule is conventional for scaled dot-product attention; it is not a universal requirement for every mechanism called attention.

Scaling does not replace numerical stabilization

Softmax implementations commonly subtract the maximum logit before exponentiation:

softmax(z)_i = exp(z_i - max(z)) / sum_j exp(z_j - max(z))

This transformation preserves the exact softmax distribution in real arithmetic while reducing overflow risk in the exponential calculation. It solves a different problem from 1 / sqrt(d_k) scaling.

Maximum subtraction addresses numerical range during softmax evaluation. Attention scaling controls the magnitude of query-key scores entering softmax. A numerically stable softmax can still produce a highly concentrated distribution when logits are far apart, and scaled logits still benefit from numerically stable evaluation.

Mixed-precision kernels may apply additional implementation-specific techniques such as higher-precision accumulation or fused reductions. Those choices affect numerical error and range, but they do not erase the semantic role of the attention scale.

Masks interact with logits after the score is formed

Causal and padding masks usually modify which key positions may receive probability mass. Implementations often represent blocked positions by adding a very negative value before softmax or by using an equivalent fused operation.

The scale and the mask therefore serve separate purposes. Scaling adjusts allowed query-key scores according to the attention definition. Masking removes disallowed relationships from the probability distribution.

The exact order can be fused inside a kernel, so source code need not expose separate tensor operations. What matters is the resulting semantics: permitted logits use the intended score scale, while blocked positions do not participate as ordinary candidates.

Changing the scale changes a trained model’s computation

For an existing checkpoint, altering the attention denominator at inference changes the function computed by the model. The query and key projections were optimized in the context of the architecture’s scoring rule. A different scale changes softmax probabilities and therefore changes the weighted combination of values.

That does not imply every small numerical difference has a predictable output effect. It does mean the scale is part of model semantics rather than a serving-only optimization that can be freely substituted.

Implementations that optimize attention kernels can rearrange operations, fuse scaling with matrix multiplication or softmax, and use algebraically equivalent forms within their numerical tolerances. The compatibility boundary is the intended scoring rule. Preserving tensor shapes while changing the effective denominator is not the same optimization.

The practical contract is compact: identify the dimension used by each query-key dot product, preserve the checkpoint’s intended scale, and treat any extra temperature or alternative normalization as an explicit model change. That keeps head width, numerical stabilization, masking, and probability concentration from being conflated into one implementation detail.

Artificial Intelligence Attention Transformers Softmax Inference