ALiBi Adds Distance-Proportional Bias to Attention Scores

ALiBi changes an attention score before softmax rather than adding a positional vector to the token representation. For a causal transformer, a key farther behind the current query receives a larger negative offset. The offset is linear in token distance and uses a slope associated with the attention head.

A simplified score for head h can be written as:

score_h(i, j) = q_i k_j^T / sqrt(d) - m_h * (i - j)

for an allowed causal pair with j <= i and positive slope m_h. The causal mask still decides which future positions are inaccessible. ALiBi changes the relative scores among positions that remain eligible.

The bias acts in logit space

The positional term is added before softmax. That placement matters because softmax converts additive logit differences into multiplicative probability ratios.

Consider two eligible keys whose content logits are equal. If their distances from the query differ by Delta, the ALiBi contribution differs by:

-m_h * Delta

Before normalization, the ratio induced solely by this bias is proportional to:

exp(-m_h * Delta)

This does not guarantee that attention probability decreases monotonically with distance. Content logits can offset or exceed the positional penalty. ALiBi supplies a distance-dependent prior in the score surface; it does not replace content-based attention.

The same distinction applies to masking. A causal mask commonly represents forbidden positions with an effectively negative-infinite score before softmax. ALiBi uses finite distance penalties for allowed positions. Treating those two mechanisms as interchangeable would change the attention domain.

Head slopes create several distance scales

ALiBi assigns different slopes to attention heads. A larger positive slope penalizes distant keys more strongly, while a smaller slope leaves content scores comparatively less affected by distance.

That variation gives heads different positional decay scales without storing a position embedding vector for every index. The exact slope schedule is part of the model definition. A checkpoint or architecture implementation must reproduce the expected schedule; merely adding an arbitrary linear penalty does not preserve model semantics.

The slope is not an attention probability and does not need to sum to one across heads. It is a coefficient in logit space. Its numerical effect also depends on the scale of the content logits entering the same addition.

Absolute indices collapse into relative distance

For the causal form above, shifting both positions by the same amount leaves their ALiBi difference unchanged:

(i + c) - (j + c) = i - j

The bias therefore depends on relative separation rather than the absolute index itself. This property is distinct from schemes that inject an explicit vector selected by absolute position.

It also means that position handling cannot be inferred from the hidden-state tensor alone. Two tokens can have identical content representations before attention while receiving different pairwise attention biases because their distances to other positions differ.

Implementations often construct the bias from position indices, a distance matrix, or an equivalent fused kernel. Those representations are operational choices. The semantic requirement is that each eligible query-key pair receives the intended head-specific distance term.

Longer sequences extend the same arithmetic rule

A linear distance formula can be evaluated for indices beyond those present in shorter training sequences without allocating a new table entry for each additional position. That arithmetic extensibility is one reason ALiBi is relevant to context-length behavior.

It is not, by itself, a guarantee that a model will retain task quality at arbitrary sequence lengths. Longer contexts change the set of candidate keys, increase the range of bias magnitudes, and can expose behavior not constrained by the training distribution. Model quality outside the trained length regime remains an empirical property of the concrete model and workload.

For a distant key, the magnitude of the penalty grows linearly with distance:

penalty = m_h * distance

In finite precision, sufficiently negative combined logits may contribute negligible softmax mass. The point at which that occurs depends on slope, content scores, dtype, softmax implementation, and surrounding numerical details. The formula alone does not define a universal effective context length.

Cache reuse still requires consistent positions

Autoregressive serving commonly stores key and value tensors from earlier tokens in a KV cache. ALiBi does not require cached keys to carry a separate positional embedding vector, but the runtime still needs enough positional information to compute the correct distance from a new query to each cached key.

If a cache is truncated, shifted, paged, or reused, logical token positions and physical cache slots must not be confused. A key stored in slot zero is not necessarily at logical position zero. Computing distance from storage offsets after cache compaction can silently alter the bias even when the cached key and value tensors themselves are unchanged.

This boundary becomes especially relevant in streaming or prefix-reuse systems. Cache management may rearrange storage for operational reasons, while ALiBi semantics refer to the logical query-key separation expected by the model.

Bidirectional attention needs a defined distance convention

The causal expression i - j assumes j <= i. A bidirectional attention pattern does not inherit that convention automatically. An implementation may require absolute distance or another signed construction depending on the architecture.

Using the causal formula unchanged in a bidirectional layer would assign opposite signs to keys on opposite sides of the query. That is a different bias function, not a neutral extension. The attention pattern and the model’s specified positional rule must be considered together.

ALiBi therefore has a narrow architectural contract: it modifies attention logits with a head-specific function of positional separation. It does not define the causal mask, token content projection, KV-cache layout, or acceptable extrapolation range. Those boundaries remain separate parts of the model and serving system.