Vision Transformer

For a decade, “vision backbone” meant a stack of convolutions. Local filters, shared weights, translation equivariance — the inductive biases that made ImageNet tractable. Transformers had taken over NLP, but vision papers still bolted attention onto CNN stems, or replaced only a few layers, as if pixels were too spatial to trust a plain sequence model.

Dosovitskiy et al.’s punchline is deliberately blunt: cut the image into patches and treat them like words. Linearly embed each patch, prepend a class token, add position embeddings, run a standard Transformer encoder. When you pre-train that recipe at sufficient scale, large-scale data trumps the missing CNN bias.

What this paper explains

The Vision Transformer (ViT) shows that a pure Transformer encoder — the same MSA + MLP stack used in NLP — can be a competitive image classifier when the only vision-specific step is the initial patch embedding:

1. Split an $H\times W$ image into a grid of $P\times P$ patches ($P=16$ in the title’s canonical setting).
2. Flatten each patch and multiply by a learned matrix $\mathbf{E}$ to get a $D$-dimensional embedding.
3. Prepend a learnable class token (BERT-style) and add learnable 1D position embeddings.
4. Run $L$ layers of LayerNorm → multi-head self-attention → LayerNorm → MLP, with residuals.
5. Take the class token’s final state, LayerNorm it, and feed a classification head.

They study ViT-Base / Large / Huge variants, pre-train on ImageNet-21k or JFT-300M, fine-tune on mid-sized benchmarks, and compare against strong ResNet / BiT baselines under a compute-aware lens.

Prior limits

• Pixel-level attention is quadratic and huge. Attending every pixel to every pixel on a $224\times 224$ image is a non-starter; something must reduce the sequence length.
• CNN-first hybrids. Prior vision + attention work typically kept convolutional stems or hierarchical feature maps so locality stayed hard-coded.
• Inductive bias vs data. On mid-sized ImageNet-scale training without strong regularization, Transformers lag comparable ResNets by a few percentage points — exactly what you expect when locality and translation equivariance are not baked in.
• The open question. If you strip those biases down to “patches + positions,” can scale alone recover (and surpass) CNN accuracy?

The mechanism

Patchify. For patch size $P$ and channels $C$, each patch is a vector in $\mathbb{R}^{P^{2}\cdot C}$. The number of patches — and thus the Transformer sequence length — is

Smaller $P$ means a longer sequence and more compute; ViT-L/16 means the Large variant with $16\times 16$ patches.

Linear patch embedding + class token + positions. With projection $\mathbf{E}\in\mathbb{R}^{(P^{2}\cdot C)\times D}$ and position table $\mathbf{E}_{\mathrm{pos}}\in\mathbb{R}^{(N+1)\times D}$:

Encoder. Pre-norm Transformer blocks (LN before MSA/MLP, residual after):

for $\ell=1\ldots L$. The image representation is the LayerNorm’d class token:

Pre-training uses an MLP head on $\mathbf{y}$; fine-tuning replaces it with a single linear layer (zero-initialized) for the target label set.

Where the 2D bias actually lives. Self-attention is global. The paper’s own accounting: 2D structure is injected mainly when cutting patches, and again when 2D-interpolating position embeddings to fine-tune at higher resolution (same $P$, longer $N$). Learnable 1D positions were enough; more elaborate 2D embeddings did not help much in their Appendix D.4 ablations.

Model ladder (Table 1). ViT-Base: $L=12$, $D=768$, MLP $3072$, $12$ heads, $86$M params. ViT-Large: $24$ / $1024$ / $4096$ / $16$ heads / $307$M. ViT-Huge: $32$ / $1280$ / $5120$ / $16$ heads / $632$M.

Hybrid footnote. You can also form the token sequence from CNN feature-map patches (even $1\times 1$ spatial patches). The headline result, though, is that raw image patches already work when data and compute are large.

Algorithm walkthrough

1. Choose image size $(H,W)$, patch size $P$, and model width $D$ (e.g. $224\times 224$, $P=16$ → $N=196$ patches + $1$ class token).
2. Extract non-overlapping patches; flatten each to $P^{2}\cdot C$ and multiply by $\mathbf{E}$.
3. Build $\mathbf{z}_{0}$ with $\mathbf{x}_{\mathrm{class}}$ in front and $\mathbf{E}_{\mathrm{pos}}$ added.
4. For each of $L$ layers: LN → multi-head self-attention → residual → LN → GELU MLP → residual.
5. Read $\mathbf{y}=\operatorname{LN}(\mathbf{z}_{L}^{0})$; apply the classification head.
6. Fine-tune at higher resolution: keep $P$ fixed, grow $N$, bilinearly interpolate the patch position embeddings on the 2D grid, replace the head, train with SGD + momentum (their fine-tune default).

The opening figure is the same pipeline in space: an image plane fractures into a patch grid; tiles lift into a token row; the class token leads them through the encoder; a label leaves the far end.

What to notice when reading

• ViT is an interface paper as much as a model paper: keep the NLP Transformer intact; change how pixels become tokens.
• Sequence length scales as $1/P^{2}$ — the patch-size knob is a direct compute/accuracy trade.
• On small data, CNNs still win; the paper’s thesis is about the large-data regime, not a blanket “convolutions are obsolete.”
• Position embeddings learn distance structure (nearby patches often get similar embeddings) even though the architecture did not hard-code a 2D relative bias.
• Hybrid CNN→Transformer stems are optional; do not confuse them with the pure ViT headline.

Results and evidence

Numbers below are as reported by Dosovitskiy et al. — not universal constants.

Headline transfer accuracies (abstract / intro). Their best model reaches 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks, when pre-trained at scale (ImageNet-21k or JFT-300M) and fine-tuned.

Table 2 framing. ViT models pre-trained on JFT-300M outperform ResNet-based baselines on the reported datasets while taking substantially less computational resources to pre-train. ViT-L/16 on JFT beats BiT-L (same pre-train data) across their task suite; ViT-H/14 improves further on ImageNet, CIFAR-100, and VTAB. ViT-L/16 pre-trained on public ImageNet-21k also transfers well; they note it could be trained on a cloud TPUv3 with 8 cores in about 30 days.

Small-data caveat (intro). Trained on mid-sized ImageNet without strong regularization, ViTs land a few percentage points below ResNets of comparable size — the inductive-bias gap the rest of the paper overcomes with scale.

Scaling (Figures 3–4, Section 4.3). Large ViTs underperform BiT ResNets when pre-trained on smaller datasets, then overtake as pre-training data grows; larger ViT variants similarly overtake smaller ones once the dataset is big enough. Their slogan: large-scale training trumps inductive bias.

Controlled compute (Section 4.4). They compare architectures under pre-training compute budgets; ViT transfers strongly per FLOP relative to ResNets/hybrids in the regimes they plot — read the paper’s figures rather than treating any single FLOP number as magic.

Limitations

• Data hunger. Without large pre-training (or heavy regularization), the missing locality bias hurts.
• Quadratic attention cost. Global MSA over $N$ patches is fine at $224$–$518$ classification resolutions; naive ViT is not a free lunch for very high-resolution dense prediction (later hierarchical / windowed vision Transformers address this — outside this paper’s claims).
• Pre-train compute caveats. The authors note efficiency depends on schedule, optimizer, and regularization, not architecture alone; their BiT comparisons are careful but not the final word for every hardware stack.
• JFT-300M access. The strongest numbers use an in-house dataset; ImageNet-21k results are the reproducible public reference point.
• Classification focus. The paper is about image recognition backbones and transfer, not detection/segmentation recipes (U-Net-style decoders, etc.).

How to read the paper

1. Figure 1 + Equations (1)–(4) — the entire architecture fits on one page; make sure patch shape, $N$, class token, and pre-norm blocks click.
2. Inductive bias paragraph (Section 3.1) — the conceptual contrast with CNNs.
3. Table 1 — Base/Large/Huge and the /16 vs /14 patch naming.
4. Section 3.2 (fine-tuning / resolution) — position-embedding interpolation when $N$ grows.
5. Table 2 + Figures 3–4 — transfer results and the data-scale story.
6. Section 4.4 + appendices — compute comparisons and position-embedding ablations if you care about “why not 2D RoPE-style bias.”

Knowledge check

Keep reading

• Attention Is All You Need — the encoder stack ViT reuses almost unchanged.
• BERT — the class-token pattern ViT copies for classification.
• ResNet — the convolutional inductive-bias baseline ViT challenges at scale.
• CLIP (Radford et al., 2021) — dual-encoder contrastive training that often starts from ViT image towers.

Sources

• Dosovitskiy et al., An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale, ICLR 2021. https://arxiv.org/abs/2010.11929
• Vaswani et al., Attention Is All You Need, NeurIPS 2017. https://arxiv.org/abs/1706.03762
• Devlin et al., BERT, NAACL 2019. https://arxiv.org/abs/1810.04805
• He et al., Deep Residual Learning for Image Recognition, CVPR 2016. https://arxiv.org/abs/1512.03385
• Kolesnikov et al., Big Transfer (BiT), ECCV 2020 — primary ResNet transfer baseline family in the ViT comparisons.