Attention with Linear Biases (ALiBi) changes a causal attention score before softmax by adding a penalty whose magnitude grows with token distance. Position is therefore represented in the score path rather than by adding a positional vector to each token representation.
For a query at position i attending to a key at position j, one head can be written schematically as:
score(i, j) = q_i · k_j / sqrt(d_k) - m_h * (i - j)for causal positions j <= i. The positive slope m_h is specific to attention head h. The causal mask still prevents access to future positions; the linear term changes the relative preference among positions that remain visible.
The bias acts on logits, not token states
Absolute or sinusoidal position embeddings typically enter the residual stream by being combined with token representations. ALiBi takes a different route. Query and key vectors are produced without adding a trained position embedding for this mechanism, then a deterministic distance-dependent value is added to their attention logits.
That placement has a precise consequence. The positional term participates directly in the exponentiated quantities consumed by softmax. If two keys have equal content-based query-key scores but different distances from the query, the nearer key receives the less negative positional adjustment for a positive ALiBi slope.
The mechanism does not force the nearer key to win. A sufficiently larger content score for a distant key can overcome the penalty. ALiBi is therefore a recency-biased scoring rule, not a hard local-attention mask.
Head-specific slopes create several distance scales
ALiBi does not normally assign one common slope to every attention head. The original construction uses a fixed set of slopes, giving different heads different rates of decay with distance.
For two visible keys separated from a query by distances d1 and d2, the positional contribution to their logit difference in head h is:
delta_bias = -m_h * (d2 - d1)A larger m_h makes distance differences matter more strongly. A smaller m_h preserves more of the content-score difference across long separations. Multiple slopes let the attention layer contain heads with different positional preferences without storing a trainable embedding for every absolute position.
The slopes are architectural constants in the original method. Treating them as arbitrary runtime temperature controls changes the scoring rule and need not preserve the behavior of a checkpoint trained with a particular slope schedule.
Translation invariance follows from relative distance
For causal self-attention, the ALiBi term depends on i - j. If both positions are shifted by the same offset c, then:
(i + c) - (j + c) = i - jso the positional bias is unchanged. This differs from a table of trained absolute position embeddings, where a token moved from one absolute index to another can receive a different position vector.
Relative-distance dependence removes the need to look up a newly trained embedding merely because an inference sequence reaches an unseen absolute index. That property is one part of ALiBi’s length-extrapolation behavior, but it is not a universal guarantee that arbitrary sequence lengths retain the same quality.
Longer contexts increase the magnitude of the penalty
The same linear rule that avoids an absolute embedding table also means that larger distances produce increasingly negative logits. For a fixed head slope:
bias(d) = -m_h * dDoubling d doubles the magnitude of this positional term. Softmax then converts logit gaps into multiplicative differences in unnormalized attention weight.
This creates a real numerical and modeling boundary. Very distant positions can receive extremely small attention probabilities in heads with steeper slopes. The architecture permits evaluation at positions beyond the training length, but permitted indexing and useful long-range retrieval are separate properties.
Recent analysis has also reported floating-point underflow effects for sufficiently negative ALiBi-biased attention values in some settings. That is an implementation and numerical concern layered on top of the mathematical linear bias; it should not be confused with a protocol-like guarantee of the original architecture.
Causal masking and ALiBi solve different constraints
The causal mask defines visibility. ALiBi ranks visible positions with an added distance-dependent preference. These operations should remain conceptually separate.
A masked future position is excluded regardless of its ALiBi value. For an allowed past position, the ALiBi penalty modifies the score but does not prohibit attention. Implementations may fuse masking, scaling, bias addition, and softmax into one kernel, yet the semantic roles remain distinct.
This separation also matters when adapting the idea outside standard causal decoder attention. A bidirectional attention pattern has different distance symmetry and visibility requirements, so copying a causal bias matrix without specifying those semantics can produce a different mechanism.
Length extrapolation is empirical, not unlimited
The original ALiBi work evaluated language models at sequence lengths beyond their training length and reported improved extrapolation relative to the positional baselines used in that study. The mechanism makes such evaluation structurally possible without extending a trained absolute-position table.
That result does not establish an infinite context window. Attention computation, model training distribution, finite precision, content retrieval, memory capacity, and implementation limits still constrain useful context. A model can accept a longer sequence while assigning negligible effective attention to information far from the current query.
The architectural claim is narrower: ALiBi injects position through deterministic relative-distance penalties on attention logits, with head-specific slopes, rather than through an absolute positional embedding table. This makes positional scoring defined at distances beyond those present during training while retaining a built-in preference whose strength grows linearly with separation.