Differential Transformer

Transformer attention is a Softmax over every past key. That Softmax is great at ranking — and equally great at leaking mass onto tokens that do not matter. When the haystack is long, irrelevant context is not a rounding error; it is a chronic distractor for retrieval, in-context learning, and grounded generation.

Differential Transformer’s punchline is almost acoustic: treat attention noise like common-mode interference. Compute two Softmax attention maps from paired query/key projections, then take their difference. Shared noise cancels; sparse peaks on the relevant span survive.

What this paper explains

Ye et al. replace the usual attention operator inside an otherwise LLaMA-style stack (RMSNorm, SwiGLU, RoPE) with differential attention:

1. Project $X$ into paired queries/keys $[Q_1; Q_2]$, $[K_1; K_2]$ and a shared value $V$.
2. Form two Softmax maps $A_1$ and $A_2$.
3. Subtract with a scalar $\lambda$: attention becomes $(A_1 - \lambda A_2) V$.
4. Multi-head DiffAttn applies GroupNorm per head and a $(1-\lambda_{\mathrm{init}})$ multiplier so early training stays stable.

They then show the architecture is not a curiosity: on matched LLaMA-style training it improves language-modeling efficiency, long-context NLL, multi-needle retrieval, in-context learning robustness, and contextual hallucination metrics — while also shrinking activation outliers in attention logits.

Prior limits

• Single Softmax. One map must both elevate the answer and suppress distractors. Softmax never goes negative, so “canceling” noise means hoping competing scores stay small — not subtracting them.
• Long distractors. Needle-in-a-haystack and multi-needle setups expose the failure mode: accuracy falls as more distracting needles appear, even when the answer span is present.
• Downstream symptoms. The paper ties diffuse attention to contextual hallucination (summarization / QA) and to brittle in-context learning under example-order permutations.
• Activation outliers. Large attention-logit spikes complicate low-bit kernels; a mechanism that reduces outliers is operationally useful beyond accuracy tables.

The mechanism

Differential attention. Split the usual $Q$ and $K$ projections into two halves (same total projection width as a baseline head budget). With head dimension $d$:

The scalar $\lambda$ is learned, parameterized so it stays near a layer-wise initialization:

with

for layer index $l$ (1-based). Early layers start with larger $\lambda_{\mathrm{init}}$; deeper layers anneal toward $0.2$.

Why subtraction helps. If $A_1$ and $A_2$ both put mass on the same irrelevant tokens, those coordinates shrink in $A_1-\lambda A_2$. Coordinates where $A_1$ is large and $A_2$ is small become the sparse peaks. The paper’s headphone analogy is apt: the second Softmax is the “noise mic.”

Multi-head wrapper. Heads share $\lambda$ within a layer. Each head output is GroupNormalized and scaled by $(1-\lambda_{\mathrm{init}})$ before concatenation and $W^{O}$ — matching the paper’s Equations (3)–(5) residual block with SwiGLU.

Algorithm walkthrough

1. Choose head count $h$ and differential head dimension $d$ (paper often uses fewer heads than Transformer at matched parameter count because each Diff head carries two Softmax maps).
2. For each head, form $Q_1,Q_2,K_1,K_2,V$ from $X$.
3. Compute $A_1=\mathrm{softmax}(Q_1 K_1^{\top}/\sqrt{d})$ and $A_2=\mathrm{softmax}(Q_2 K_2^{\top}/\sqrt{d})$.
4. Form $(A_1-\lambda A_2)V$; apply GroupNorm and the $(1-\lambda_{\mathrm{init}})$ scale.
5. Concatenate heads, project with $W^{O}$, residual + MLP as in LLaMA.
6. Optionally implement the two Softmax paths with FlashAttention-style kernels (Appendix A) — DiffAttn is not anti-FlashAttention; it reuses it twice then subtracts.

The opening figure shows the same story spatially: two Softmax heat strips enter a subtractor; noise bars cancel; a sparse answer peak leaves toward $V$.

What to notice when reading

• DiffAttn changes the score algebra, not the KV cache policy (contrast StreamingLLM’s sink/window cache).
• $\lambda$ is a layer scalar with a carefully chosen $\lambda_{\mathrm{init}}$ schedule — ablations with fixed $0.8$ / $0.5$ still work, but the schedule is the default.
• GroupNorm is load-bearing in their 1.4B ablations: removing it hurts validation loss.
• Attention scores can be negative after subtraction; the operator is no longer a convex combination of values.
• Throughput overhead in their microbenchmarks is modest (roughly $5$–$12\%$ slower than matched Transformer at 3B/13B in Table 7 of the paper), not a $2\times$ tax.

Results and evidence

All numbers below are as reported by Ye et al. on their matched training setups — not universal constants.

LM Eval Harness (3B, 1T tokens training comparison in their Table 1). Diff-3B averages 60.6 across ARC-C/E, BoolQ, HellaSwag, OBQA, PIQA, WinoGrande vs 57.5 OpenLLaMA-3B-v2 and 56.8 StableLM-base-alpha-3B-v2 under the authors’ reported comparison.

Scaling (Section 3.2). Their scaling curves claim Diff Transformer needs only about 65% of the model size or training tokens for Transformer to match language-modeling loss.

Multi-needle retrieval at 4K (Table 2). With $N$ answer-relevant needles and distractors, average accuracy stays higher for Diff as $N$ grows — e.g. at $N=6$, 0.85 vs Transformer 0.55. They also report a case where the accuracy gap reaches about 30%.

Attention mass on answer vs noise (Table 3). In the retrieval probe, Diff assigns much more attention to answer spans (e.g. 0.27–0.40 across depths) and far less to noise (~0.01–0.02) than Transformer (~0.03–0.09 answer, ~0.49–0.54 noise).

Contextual hallucination (Tables 4–5 in paper; summarization / QA hit rates). Diff improves the authors’ hallucination-oriented metrics on XSum / CNN-DM / MultiNews and on Qasper / HotpotQA / 2WikiMQA versus their Transformer baseline (e.g. MultiNews 0.61 vs 0.42; Qasper 0.39 vs 0.28).

Activation outliers (Table 5). Top attention-logit magnitudes drop sharply (Transformer top-1 318.0 vs Diff 38.8 in their measurement), which they flag as helpful for low-bit attention kernels.

Limitations

Compressed from the paper’s scope and what it does not claim:

• Gains are measured against the authors’ LLaMA-style Transformer reimplementation and selected public 3B checkpoints — not against every 2024–2025 long-context baseline.
• Differential heads change FLOPs/memory shape (two Softmax maps); their throughput table shows a small slowdown, not a free lunch.
• Negative attention weights change the usual probabilistic interpretation of attention; interpretability tools that assume a simplex may need care.
• Sparse patterns suggest KV compression opportunities, but the paper leaves production sparse-KV serving as future work.
• Hallucination metrics are task-specific automatic scores, not a universal factuality guarantee.

How to read the paper

1. Abstract + §1 — noise-canceling motivation.
2. §2.1 — DiffAttn equations (1)–(2) and $\lambda$ parameterization.
3. §2.1–2.2 — multi-head GroupNorm wrapper and the full block.
4. §3.2 — scaling curves (~65% claim).
5. §3.4 + Tables 2–3 — multi-needle accuracy and attention-to-answer vs noise.
6. §3.6–3.7 — hallucination metrics and outlier magnitudes.
7. Appendix A — FlashAttention-friendly implementation sketch.

Knowledge check

Keep reading

1. Original paper (arXiv:2410.05258) — §2 mechanism, §3.4 retrieval, Appendix A kernels.
2. Related Fanout Daily: Attention is All You Need, FlashAttention, StreamingLLM, GQA, RoPE.
3. Vaswani et al., Attention Is All You Need — arXiv:1706.03762.
4. Dao et al., FlashAttention — arXiv:2205.14135.

Sources

• Ye, Dong, Xia, Sun, Zhu, Huang, Wei, Differential Transformer, 2024 — arXiv:2410.05258
• Vaswani et al., Attention Is All You Need, 2017 — arXiv:1706.03762
• Touvron et al., LLaMA, 2023 — arXiv:2302.13971
• Dao et al., FlashAttention, 2022 — arXiv:2205.14135
• Su et al., RoFormer (RoPE) — arXiv:2104.09864