ALiBi Adds Distance Penalties Directly to Attention Logits

Thu, 24 Sep 2026 00:00:00 +0000

Attention with Linear Biases (ALiBi) changes a causal attention score before softmax by adding a penalty whose magnitude grows with token distance. Position is therefore represented in the score path rather than by adding a positional vector to each token representation.

For a query at position i attending to a key at position j, one head can be written schematically as:

copied = false, 1600)" class="absolute right-3 top-3 z-10 rounded-lg border border-line-strong bg-surface px-2 py-1 font-mono text-[0.8rem] text-muted opacity-0 transition group-hover:opacity-100 hover:bg-ink hover:text-white dark:border-night-line dark:bg-night dark:text-night-muted dark:hover:bg-white dark:hover:text-ink">

score(i, j) = q_i · k_j / sqrt(d_k) - m_h * (i - j)

for causal positions j <= i. The positive slope m_h is specific to attention head h. The causal mask still prevents access to future positions; the linear term changes the relative preference among positions that remain visible.

ALiBi Adds Distance-Proportional Bias to Attention Scores

Thu, 24 Sep 2026 00:00:00 +0000

ALiBi changes an attention score before softmax rather than adding a positional vector to the token representation. For a causal transformer, a key farther behind the current query receives a larger negative offset. The offset is linear in token distance and uses a slope associated with the attention head.

A simplified score for head h can be written as:

score_h(i, j) = q_i k_j^T / sqrt(d) - m_h * (i - j)

for an allowed causal pair with j <= i and positive slope m_h. The causal mask still decides which future positions are inaccessible. ALiBi changes the relative scores among positions that remain eligible.

ALiBi on Nalar

ALiBi Adds Distance Penalties Directly to Attention Logits

ALiBi Adds Distance-Proportional Bias to Attention Scores