ALiBi: Attention with Linear Biases for Length Extrapolation

When you train a Transformer language model, you pick a training length $L$. At inference you often want more context than $L$. Vaswani et al. hoped Transformers might “extrapolate to sequence lengths longer than the ones encountered during training.” In practice, models that inject position embeddings into the token stream — sinusoidal, rotary, or learned relatives like T5’s bias — often lose perplexity as soon as validation length $L_{\mathrm{valid}}$ exceeds $L$.

Attention with Linear Biases (ALiBi) throws out bottom-of-network positional embeddings. Instead it adds a fixed, head-specific linear penalty to the attention scores: farther keys get a more negative bias before Softmax. The slope $m$ is set once (not learned) and reused across domains. The punchline in the abstract: a 1.3B model trained on length 1024 extrapolates to 2048, matching a sinusoidal model trained on 2048 while training 11% faster and using 11% less memory (paper-reported).

What this paper explains

Press, Smith, and Lewis ask a systems-flavored modeling question with a tiny architectural answer:

1. Define extrapolation. Keep next-token language modeling quality as $L_{\mathrm{valid}}$ grows past the training length $L$.
2. Show the failure mode. On WikiText-103, sinusoidal / rotary / T5 relative methods degrade as $L_{\mathrm{valid}}$ increases (their Figure 1; appendix Tables 2–4).
3. Replace PE with score bias. For query $i$, Softmax sees

with head-specific constant $m$.

1. Pick slopes once. For $n$ heads, use the geometric progression starting at $2^{-8/n}$ with the same ratio — e.g. eight heads use $\tfrac{1}{2},\tfrac{1}{4},\ldots,\tfrac{1}{256}$.
2. Scale the story. Beyond WikiText-103, transfer the same slopes to BooksCorpus and to a 1.3B model on CC100+RoBERTa, training shorter than the sinusoidal baseline while matching or beating its extrapolated perplexity.

This is not Ring Attention (sequence-parallel KV circulation) and not a KV-cache eviction policy. It is a position inductive bias sitting inside the attention score matrix.

Prior limits

• Sinusoidal / absolute PE (Vaswani et al.) bind training and inference lengths in practice: when $L_{\mathrm{valid}} > L$, WikiText curves in the paper climb in perplexity even under sliding-window evaluation.
• Rotary (RoPE) and T5 relative bias improve relative modeling, but the authors’ WikiText extrapolation plots still degrade past $L$ (T5 also OOMs on their 32GB GPU at longer eval lengths in Figure 1).
• Training longer is expensive. Appendix Figure 7 shows words/sec falling as $L$ grows — so “just train at the inference length” is a budget decision, not free.
• Early RNNs routinely trained short and assumed longer eval; Transformers lost that habit once PE entered the bottom of the stack.

The mechanism

Drop bottom PE. Token embeddings enter the stack without added position vectors.

Bias the scores, not the vectors. After $\mathbf{q}_i\mathbf{K}^{\top}$ (and the usual scale), add a vector whose $j$-th entry is $-m\cdot(i-j)$ for keys to the left of the query (causal LM). Softmax then sees a linear distance penalty. Everything else — FFN, residual path, causal mask — is unchanged.

Head-specific slopes, frozen. Different heads get different $m$ so the layer keeps a mix of near-sighted and longer-range heads. Values are not trained; they are the geometric recipe above. The paper argues you can reuse the same recipe across domains and sizes (WikiText → Books → 1.3B), analogous to how sinusoidal wavelengths were set once.

Recency inductive bias. Larger distance ⇒ more negative logit ⇒ Softmax mass shifts toward recent keys. That is intentional: language modeling often rewards local context, and the bias stays well-defined for any $i$ larger than those seen in training.

Algorithm / figure walkthrough

1. Choose head count $n$; set slopes $m_h = 2^{-8/n}\cdot(2^{-8/n})^{h}$ for $h=0,\ldots,n-1$ (paper’s geometric construction).
2. Embed tokens without absolute / rotary PE at the bottom.
3. For each attention head, compute scores $\mathbf{q}_i\mathbf{K}^{\top}/\sqrt{d}$ as usual.
4. Add $m_h\cdot[-(i-1),\ldots,0]$ to those scores (causal positions only).
5. Softmax → attend → continue the Transformer block.
6. At eval, feed sequences with $L_{\mathrm{valid}} > L$; the bias formula does not need new PE rows.

The opening visual is the same arc: ENTER with PE that breaks past $L$; RUN with frozen slope × distance on the logits; LEAVE with usable longer context without retraining at that length.

What to notice when reading

• Extrapolate ≠ memorize longer PE tables. ALiBi never allocates a longer embedding — the bias is a closed-form function of distance.
• Where the non-linearity lives. Softmax still mixes keys; ALiBi only reshapes the logits. Do not confuse the linear bias with linear attention.
• Comparison fairness. On the 1.3B runs, ALiBi trains on shorter $L$ than the sinusoidal baseline it is compared against when claiming speed/memory wins (Figure 5 / Table 11 narrative).
• Non-extrapolating regime. On the large corpus, when both methods train at the same $L$, ALiBi is merely similar to sinusoidal (Appendix Table 12) — the headline advantage is the train-short / eval-long setting plus WikiText wins even without extrapolation.
• Implementation cost. Authors claim negligible memory overhead vs sinusoidal at the same $L$ (about 0–0.7%) and no extra parameters — you can implement by changing a few lines in the attention kernel.

Results and evidence

Numbers below are from the paper (arXiv:2108.12409) — not re-measured here.

WikiText-103 extrapolation (Figure 1; Tables 2–4). Models trained at $L\in\{512,1024,3072\}$: sinusoidal / rotary / T5 degrade as $L_{\mathrm{valid}}$ grows; ALiBi stays flat or improves over a wide range past $L$. Example at train $L=512$: sinusoidal PPL goes from 20.05 at 512 inputs toward the low-20s+ as length increases, while ALiBi starts at 19.73 and continues improving into the high-19s in the tabulated range (see Table 2 for the full grid).

BooksCorpus transfer. Same slope recipe, BERT vocabulary, tied embeddings — ALiBi still extrapolates where sinusoidal does not (paper §4.1).

1.3B on CC100+RoBERTa (461 GB mix; 25 layers, 16 heads, $d=2048$). Figure 5 / abstract:

Limitations

Compressed from the paper’s discussion and experimental scope:

• Language modeling focus. Extrapolation is measured with perplexity on LM corpora; the paper does not claim universal wins on every downstream task or every architecture family.
• Finite stretch, not unbounded. ALiBi curves stay healthy far past $L$ in their plots, but the authors do not assert arbitrary context without eventual degradation.
• Recency bias is a prior. Tasks that need sharp long-range links may want different position machinery (or complementary systems tricks).
• Parallel work. Wennberg & Henter (2021) also bias attention by distance with an RBF and trainable parameters — different functional form and cost profile.
• Venue string. arXiv abs/OpenAlex list this as a 2021 preprint; we do not invent a conference badge beyond what those sources show.

How to read the paper

1. Abstract + §1 — extrapolation definition and Figure 1 failure of PE methods.
2. §3 — the Softmax bias formula, slope recipe, Figure 3.
3. §4.1 — WikiText / Books results and ablations.
4. §4.2 — 1.3B CC100+RoBERTa Figure 5 and memory/speed claims.
5. §5 related work + appendix tables for exact PPL grids.

Knowledge check

Keep reading / Sources

• Paper: arXiv:2108.12409 — Press, Smith, Lewis, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• Code & models (authors): github.com/ofirpress/attention_with_linear_biases
• Contrast in Fanout Daily: RoPE, Ring Attention, StreamingLLM
• Related position work cited by the authors: Vaswani et al. 2017 (sinusoidal); Su et al. (RoPE); Raffel et al. (T5 relative bias); Wennberg & Henter 2021 (RBF distance bias)