Attention Is All You Need

Sequence models before 2017 were mostly recurrent or convolutional. RNNs give a natural left-to-right state, but that also forces sequential computation: you cannot finish position $t$ before $t-1$. Convolutions parallelize better, yet stacking enough layers to connect distant tokens still costs depth. The Transformer’s punchline is blunt: drop recurrence and convolutions for transduction, and let attention build every pairwise dependency in one parallel step.

The architecture is an encoder–decoder stack. Inside each layer, multi-head self-attention mixes tokens; feed-forward networks process each position independently; residual connections and layer norm stabilize depth. Sinusoidal positional encodings inject order because attention alone is permutation-equivariant.

What this paper explains

Vaswani et al. introduce the Transformer as a transduction model built only from attention and position-wise MLPs. Scaled dot-product attention is:

The $\sqrt{d_k}$ scale keeps dot products from growing with dimension so softmax stays out of the saturated regime. Multi-head attention runs $h$ such maps in parallel with learned projections, then concatenates:

where $\mathrm{head}_i = \mathrm{Attention}(QW_i^{Q}, KW_i^{K}, VW_i^{V})$.

They also show encoder–decoder attention (queries from the decoder, keys/values from the encoder) and masked decoder self-attention so generation stays causal.

Prior limits

• RNNs / LSTMs / GRUs — expressive sequential state, but $O(n)$ sequential steps and long paths hurt distant dependencies and hardware utilization.
• Convolutional seq2seq — more parallel, but large receptive fields need many layers or dilated stacks.
• Earlier attention (Bahdanau, Luong) — usually an add-on to an RNN encoder–decoder, not a replacement for recurrence.

The mechanism

Self-attention builds a weighted average of value vectors. For each query position, scores against all keys decide how much of each value to mix in. Because every position talks to every other in one matmul, the graph diameter collapses.

Multi-head attention is the capacity trick: different heads can specialize (syntax vs longer-range cues) while keeping per-head $d_k = d_{\mathrm{model}}/h$ small enough that the scale factor works.

Positional encodings use fixed sinusoids of different frequencies so relative offsets are linearly recoverable; the paper also notes learned embeddings work similarly on their WMT setup.

Algorithm / figure walkthrough

1. Embed tokens and add positional encodings → $d_{\mathrm{model}}$ vectors.
2. Encoder layer: multi-head self-attention → residual + norm → position-wise FFN → residual + norm. Stack $N=6$ (base).
3. Decoder layer: masked self-attention → encoder–decoder attention → FFN, each with residuals/norms.
4. Linear + softmax projects to vocabulary probabilities.

Base config they highlight: $d_{\mathrm{model}}=512$, $d_{\mathrm{ff}}=2048$, $h=8$, $d_k=d_v=64$. Big: $d_{\mathrm{model}}=1024$, $h=16$, more layers/wider FFN.

The opening visual contrasts a long RNN path with a single attention hop between distant tokens.

What to notice when reading

• Why they scale by $\sqrt{d_k}$ — not a cosmetic constant.
• How masking in the decoder enforces causality without an RNN clock.
• Table 1’s complexity comparison: attention is $O(n^{2}\cdot d)$ per layer but $O(1)$ sequential; that trade mattered once GPUs were the bottleneck.

Results and evidence

On WMT 2014 English→German, Transformer (big) reports 28.4 BLEU, beating prior best published results including ensembles at the time. On English→French, big reaches 41.8 BLEU. The base model already hits 27.3 BLEU on En→De with far less training cost than many deep RNN systems.

They also show that more heads help up to a point (8 heads in base), and that removing positional encoding hurts — order is not free once recurrence is gone.

Limitations

Stated or implied by the setting:

• Quadratic attention in $n$ — fine for sentence pairs; painful for very long context (later work: sparse attention, FlashAttention, etc.).
• Autoregressive decoding still generates one token at a time; encode-side parallelism does not remove decode latency.
• WMT-centric evaluation; the architecture generalized far beyond MT, but that is later history, not this paper’s claim.

How to read the paper

1. Abstract + §1–2 — motivation and related work.
2. §3.1–3.3 — attention, multi-head, architecture diagram (Figure 1).
3. §3.2.1 — scaled dot-product vs additive attention.
4. §5 — WMT results and ablations (Table 3).
5. §4 — why path length and parallelization matter.

Knowledge check

Keep reading

1. Original paper (arXiv:1706.03762) — Figure 1, §3, Table 2–3.
2. Related Fanout Daily: FlashAttention, GQA, BERT, PagedAttention.

Sources

• Vaswani et al., Attention Is All You Need, NeurIPS 2017 — arXiv:1706.03762