RoFormer: Enhanced Transformer with Rotary Position Embedding

Transformers need to know where a token sits. Absolute sinusoidal embeddings add a position vector to each token; relative schemes try to make attention depend on the offset between tokens, but many of those tricks add position into the content stream in ways that break linear / kernelized attention.

RoPE’s punchline is geometric. Treat each even/odd pair of dimensions as a 2-D plane. At position $m$, rotate the query (and key) by angle $m\theta$. Then the attention score between positions $m$ and $n$ depends on the relative angle $(m-n)\theta$ — absolute indices on each side, relative structure in the product.

What this paper explains

Position information in PLMs is usually injected in one of three ways:

1. Absolute embeddings (sinusoidal or learned) added to token embeddings.
2. Relative biases or embeddings that modify attention logits using $m-n$.
3. Hybrids that try to keep both.

Su et al. formulate a functional constraint: after position-aware transforms $f_q, f_k$, the inner product $\langle f_q(x_m, m), f_k(x_n, n)\rangle$ should be a function of the content and the relative offset $m-n$. They then exhibit a solution: multiply $W_q x_m$ and $W_k x_n$ by rotation matrices indexed by $m$ and $n$.

The paper asks:

1. Can relative position enter self-attention through a multiplicative (rotation) path instead of an additive one?
2. Does that form keep sequence-length flexibility and a long-term decay intuition?
3. Can the same idea attach to linear attention, where additive relative methods struggle?

Prior limits

• Additive absolute PE (Vaswani et al.) works, but position and content are mixed in the same vector before $W_q, W_k$. Relative structure is only implicit.
• Relative position methods (Shaw et al., Transformer-XL, DeBERTa-style biases, etc.) often add position-specific terms into attention. The paper notes that adding position into the context representation can make those ideas awkward for linear self-attention.
• Complex-space or other PE variants help empirically but still commonly follow the “add position to content” pattern the authors want to avoid.
• Practitioners want length extrapolation and decaying influence with distance — properties that need to fall out of the formulation, not only from training data.

The mechanism

Constraint: relative in the product

Write queries and keys as position-aware maps of content $x$. The authors require that the attention affinity depend on $x_m$, $x_n$, and $m-n$. In the 2-D case they solve this with complex multiplication (rotation):

The product then involves $e^{i(m-n)\theta}$ — relative phase — while each side only needed its own absolute index.

Matrix view (same geometry)

Equivalently, with real 2-D vectors,

Rotating $q$ by $m\theta$ and $k$ by $n\theta$ leaves the inner product equal to rotating one vector by $(m-n)\theta$ against the other.

General even dimension $d$

Split $\mathbb{R}^d$ into $d/2$ planes and apply a block-diagonal rotation:

with pairwise angles $\theta_i = 10000^{-2i/d}$ (same base as the Transformer sinusoids). Implementation never materializes the huge matrix: rotate each $(x_{2i}, x_{2i+1})$ pair in place.

Properties the paper highlights

• Relative by construction — absolute rotations compose into an $m-n$ dependency in attention.
• Length flexibility — positions are angles; you can evaluate at indices not seen in training (with the usual caveats).
• Long-term decay — with the $\theta_i$ schedule, the expected contribution of distant pairs tends to fade as $|m-n|$ grows (their §3.3 / §3.4.3 intuition).
• Linear attention compatible — because position is a rotation of $q$ and $k$, not an additive term glued onto the value pathway the way some relative methods need.

Algorithm walkthrough

1. Project tokens to $q_m = W_q x_m$, $k_n = W_k x_n$ as usual.
2. Choose even head dimension $d$; set $\theta_i = 10000^{-2i/d}$ for $i = 0,\ldots,d/2-1$.
3. For each position index $m$ and each pair $i$, rotate $(q_{m,2i}, q_{m,2i+1})$ by $m\theta_i$ (same for keys with $n\theta_i$).
4. Compute attention with the rotated $q', k'$ — e.g. $\mathrm{softmax}(q' k'^\top / \sqrt{d})\, v$ (values are typically not rotated in the standard RoPE recipe).
5. Stack as in any Transformer; the paper’s RoFormer is “Transformer + RoPE” evaluated on MT, MLM-style pretraining, GLUE, and a Performer-style linear-attention check.

The opening visual shows additive PE as a vertical stack of content+position versus RoPE as a turn of the query/key arrows. The slider above lets you change $m$ and watch a 2-D pair rotate while the relative angle to a fixed key updates.

What to notice when reading

• The derivation order: constraint on $\langle f_q, f_k\rangle$ first, rotation solution second — not “we tried a rotation and it worked.”
• Values are left unrotated in the usual formulation; position enters the routing (scores), not the value vectors themselves.
• The $\theta_i$ schedule is doing real work for the decay story; it is not an arbitrary constant.
• Their linear-attention experiment is about compatibility of the PE form, not a claim that Performer+RoPE is SOTA everywhere.

Results and evidence

Numbers below are from the paper’s reported tables (arXiv:2104.09864), not independent re-runs.

Machine translation (Table 1). On WMT 2014 English→German, under their fairseq setup (joint 37k BPE, Adam $\beta=(0.9,0.98)$, beam 4, length penalty 0.6):

Language modeling / downstream. They replace BERT’s sinusoidal PE with RoPE during MLM pretraining (BookCorpus + Wikipedia split) and fine-tune on GLUE; the narrative claim is that RoFormer consistently overcomes the sinusoidal baseline under their settings (see paper §4.2–4.3 for per-task scores — report from the PDF rather than inventing cells here).

Linear attention. §4.4 pairs RoPE with Performer-style linear attention to argue the multiplicative PE form still applies when softmax attention is replaced.

Chinese data. §4.5 adds further classification-style checks beyond the English GLUE suite.

Limitations

Compressed from the paper’s setting and from what it does not claim:

• Empirical lifts vs strong absolute PE can be modest on short-sequence tasks (Table 1’s +0.2 BLEU).
• Long-context extrapolation later became a whole research area (NTK-aware scales, YaRN, ALiBi comparisons, etc.); this paper supplies the mechanism and early evidence, not the final 2024–2026 length-extrapolation cookbook.
• Most experiments are encoder or encoder–decoder style PLMs of their period; decoder-only LLM training recipes post-date the work.
• Theoretical decay arguments are about the PE form’s inductive bias; they are not a guarantee for every downstream metric.

How to read the paper

1. Abstract + §1 — relative-in-the-product goal vs additive PE.
2. §3.1–3.2 — 2-D complex/rotation derivation, then block-diagonal $R^d_{\Theta,m}$.
3. §3.3 properties — length flexibility, decay, linear-attention hook.
4. §4.1 Table 1 — WMT En→De sanity check.
5. §4.2–4.4 — pretraining/GLUE + Performer experiment.
6. §5 — claims to keep separate from later industrial RoPE folklore.

Knowledge check

Keep reading

1. Original paper (arXiv:2104.09864) — §3 derivation, Figure 1, Table 1.
2. Vaswani et al., Attention Is All You Need — sinusoidal absolute PE — arXiv:1706.03762.
3. Press et al., ALiBi — alternative relative bias for length extrapolation — arXiv:2108.12409.
4. Related Fanout Daily: Attention Is All You Need, GQA, FlashAttention.

Sources

• Su, Lu, Pan, Murtadha, Wen, Liu, RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021 — arXiv:2104.09864
• Vaswani et al., Attention Is All You Need, NeurIPS 2017 — arXiv:1706.03762
• Press, Smith, Lewis, Train Short, Test Long: Attention with Linear Biases… (ALiBi), 2021 — arXiv:2108.12409