Skip to main content
fanout
Archive
issue 2026-08-16AI Research42 minICML 2021 / arXiv 2021interactive

CLIP: Learning Transferable Visual Models From Natural Language Supervision

This research paper shows how natural-language supervision scales vision: contrastive dual encoders on 400M image–text pairs learn a shared embedding space you can query with text at test time.

Radford et al. · Original paper

CLIP aligns image and text towers with a batch similarity matrixLeft: matched image–text pairs enter dual encoders. Middle: contrastive training lifts the diagonal of the N by N similarity matrix. Right: class-name prompts leave as a zero-shot classifier.ENTER · PAIRSIMAGE · TEXTIMGf(x)TXTg(y)N MATCHED EXAMPLESRUN · CONTRASTLIFT THE DIAGONALSYMMETRIC InfoNCE / τLEAVE · ZERO-SHOTTEXT → CLASSIFIERCLSPROMPTSargmaxI·TNO TASK LABELS NEEDEDInsight: contrast batch pairings into a shared space — then query it with class-name text.CLIP dual-encoder contrast on mobileStacked cards: paired image–text enter, contrastive matrix run, zero-shot classifier leave.CLIP · LANGUAGE SUPERVISIONENTER · IMAGE / TEXT PAIRSDual encoders f and gRUN · N×N CONTRASTLift diagonal · suppress mismatchesLEAVE · ZERO-SHOTEmbed prompts · score I·TShared space beats closed-set ImageNet heads alone

Supervised ImageNet pre-training taught a generation of vision models — and also boxed them in. Every new task needed a new labeled set; every closed label space was a wall. CLIP asks a different question: what if the supervision signal is natural language that already co-occurs with images on the internet, and the learning problem is not “predict the caption word-by-word” but “decide which caption goes with which image in a batch”?

The punchline is a dual-encoder contrastive recipe. An image encoder and a text encoder map a batch of NN pairs into a shared embedding space. Cosine similarities form an N×NN \times N matrix. Training maximizes the matched diagonal and suppresses the N2−NN^2 - N mismatches with a symmetric cross-entropy (InfoNCE / multi-class N-pair) loss. After pre-training on 400 million (image, text) pairs (WIT), you freeze the towers and, at test time, embed class names with the text tower to synthesize a zero-shot linear classifier — matching the original ResNet-50’s ImageNet accuracy in their headline comparison without using any of ImageNet’s 1.28M training labels.

What this paper explains

Three ideas sit stacked:

  1. Natural language as supervision — captions, alt text, and co-occurring phrases already name visual concepts at internet scale.
  2. Contrastive dual encoders — cheaper and stronger than generative caption prediction for representation learning (their Figure 2 ablation: bag-of-words prediction is already better than a Transformer language-model baseline; swapping to CLIP’s contrastive objective adds another roughly 4× efficiency gain on zero-shot ImageNet transfer rate).
  3. Zero-shot transfer — the text encoder is a hypernetwork that emits classifier weights from class names (or prompts), so one pre-trained model covers many datasets without dataset-specific training heads.

They also scale both ResNet and Vision Transformer image towers, train for 32 epochs with a huge minibatch (32,768), and evaluate zero-shot and linear-probe transfer across a broad suite of vision tasks.

CLIP dual encoders and similarity matrixImage encoder f and text encoder g embed a batch. Cosine similarities form an N by N matrix whose diagonal are the matched pairs.fimageI₁…IₙgtextT₁…TₙSᵢⱼ = (Iᵢ · Tⱼ) / τ — diagonal = matches
Dual towers embed the batch; contrastive loss lifts matched similarities on the diagonal.

Prior limits

  • Closed-set classifiers need a labeled training set per dataset; they do not invent new categories at test time.
  • Early language-supervised vision (Visual N-Grams and related) showed the zero-shot idea was possible but weak — their Table 1 cites Visual N-Grams at 11.5% ImageNet accuracy versus CLIP’s best 76.2%.
  • Generative / predictive caption objectives try to emit the exact words of co-occurring text. That target is noisy and expensive; CLIP’s authors emphasize that contrastive “which pair matched?” is a more scalable proxy.
  • Pure image-only SSL (SimCLR and friends) learns strong features but does not give you a text interface for naming classes without a separate head or fine-tune.

The mechanism

Encoders. Let ff be the image encoder and gg the text encoder. For a batch {(xi,yi)}i=1N\{(x_i, y_i)\}_{i=1}^{N},

Ii=f(xi)∥f(xi)∥2,Ti=g(yi)∥g(yi)∥2.I_i = \frac{f(x_i)}{\|f(x_i)\|_2}, \qquad T_i = \frac{g(y_i)}{\|g(y_i)\|_2}.

Similarity matrix. With a learnable temperature τ\tau (optimized as a log-parameterized scalar, capped in training),

Sij=Ii⋅Tjτ.S_{ij} = \frac{I_i \cdot T_j}{\tau}.

Symmetric InfoNCE. Treat each row as “which text matches this image?” and each column as “which image matches this text?”:

L=12N∑i=1N(−log⁡eSii∑jeSij−log⁡eSii∑jeSji).\mathcal{L} = \frac{1}{2N}\sum_{i=1}^{N}\Big( -\log \frac{e^{S_{ii}}}{\sum_{j} e^{S_{ij}}} -\log \frac{e^{S_{ii}}}{\sum_{j} e^{S_{ji}}} \Big).

That is the multi-class N-pair / InfoNCE construction the paper cites (Sohn 2016; Oord et al. 2018), applied symmetrically to image→text and text→image.

Interactive

Temperature sharpens the match

Toy batch of N = 4 pairs. Softmax each row of cosine similarities after dividing by τ. Lower τ peaks the matched diagonal; higher τ flattens the distribution. Teaching toy — not a live CLIP forward pass.

Mid τ: matched pairs still win, but gradients reach near-miss negatives too.

Zero-shot classification. For a downstream label set {c1,…,cK}\{c_1,\ldots,c_K\}, embed prompt strings such as “a photo of a {ck}\{c_k\}” with gg, L2-normalize, and classify an image embedding II by

k^=arg⁡max⁡k  I⋅Tck.\hat{k} = \arg\max_k \; I \cdot T_{c_k}.

Prompt engineering and ensembling (multiple prompt templates averaged in embedding space) further lift zero-shot accuracy in their §3.1.4.

Zero-shot classifier from text promptsClass-name prompts go through the text encoder to form classifier vectors. An image embedding is scored by cosine similarity and argmax.imageI = f(x)/‖·‖test imagea photo of a doga photo of a cata photo of a cara photo of a treeargmaxₖ I · Tₖzero-shot labeltext tower = classifier
Prompted class names become vectors in the same space as images — no ImageNet head required.

Algorithm walkthrough

  1. Sample a minibatch of NN image–text pairs from WIT.
  2. Encode images with ff and texts with gg; L2-normalize both.
  3. Form logits S=IT⊤/τS = IT^\top / \tau (outer product of normalized embeddings).
  4. Compute cross-entropy of SS vs. labels 0..N−10..N-1 and of S⊤S^\top vs. the same labels; average the two losses.
  5. Update ff, gg, and τ\tau with AdamW + cosine LR for 32 epochs (batch 32,768 in the released hyperparameter table).
  6. At evaluation, embed class-name prompts with gg once, then classify by cosine similarity — no task-specific training examples required for the zero-shot protocol.

The opening figure stages the same story as enter (paired examples), run (contrast the N×NN\times N matrix), leave (text-defined classifier weights).

What to notice when reading

  • Figure 1’s cartoon is the whole paper: joint towers + batch matching, then text as a classifier synthesizer.
  • Figure 2’s efficiency plot is the justification for contrastive over caption prediction — quote their relative rates, do not invent FLOPs.
  • Temperature τ\tau is learned, not a frozen hyperparameter — it rescales the logits that Softmax sees.
  • Zero-shot CLIP is not magic open-world perception; it is retrieval in a joint space against a finite text-defined label set you chose.
  • Their robustness section studies natural distribution shifts relative to ImageNet-trained models — read claims as paper-measured, not universal laws.

Results and evidence

Numbers below are from the paper (treat as author-reported, not timeless absolutes):

ClaimPaper figure
Pre-training scale400M (image, text) pairs (WIT)
Zero-shot ImageNet (best CLIP)76.2% top-1; 95% top-5 (matches Inception-V4 top-5 in their text)
Visual N-Grams on ImageNet11.5% (Table 1 baseline)
Headline ResNet-50 matchZero-shot CLIP matches original ResNet-50 ImageNet accuracy without ImageNet train labels
Contrastive vs predictiveContrastive objective ~4× more efficient than their bag-of-words predictive baseline on zero-shot ImageNet learning curves (Figure 2)
LM vs BoW predictiveTransformer language-model predictive objective learned ~3× slower than BoW prediction in the same ablation
Training recipe32 epochs; batch size 32,768; AdamW; cosine decay; learnable τ\tau

Linear-probe evaluations (Appendix) further show the learned image features are competitive with contemporaneous representation-learning systems when you do allow a fitted head — but the distinctive product is the text-conditioned zero-shot interface.

Limitations

Compressed from the authors’ Limitations / Broader Impacts discussion:

  • Zero-shot still fails on many fine-grained or abstract tasks; language is not a complete substitute for every labeled dataset.
  • WIT is unfiltered internet co-occurrence — models absorb social biases and dual-use risks (surveillance-related probes are discussed in §7).
  • Prompt wording matters; poor prompts understate the model’s feature quality.
  • Compute for the largest CLIP runs is substantial; the paper is honest that scale is part of the result.
  • Overlap between pre-training data and downstream test sets is analyzed; they attempt to measure contamination rather than ignore it.

How to read the paper

  1. Abstract + Figure 1 — dual towers and zero-shot classifiers.
  2. §2.3–2.5 — why contrastive, model choices, training hyperparameters (Appendix F for the batch / τ\tau table).
  3. §3.1 — zero-shot protocol, Table 1 vs Visual N-Grams, prompt engineering.
  4. §3.2–3.3 — representation learning and robustness.
  5. §6–7 — limitations and broader impacts (do not skip).

Knowledge check

In a CLIP training batch of N pairs, what does the model learn to identify?

How does CLIP build a zero-shot ImageNet classifier after pre-training?

Why did the authors prefer a contrastive objective over predicting caption text directly?

Keep reading

  1. Original paper (arXiv:2103.00020) — Figures 1–3, Table 1, §2–3.
  2. Fanout Daily: Vision Transformer — the image tower CLIP often scales.
  3. Related ideas: SimCLR (image-only contrastive), InfoNCE (Oord et al.), Visual N-Grams (Li et al. 2017).

Sources

  • Radford et al., Learning Transferable Visual Models From Natural Language Supervision, ICML 2021 — arXiv:2103.00020
  • Oord et al., Representation Learning with Contrastive Predictive Coding (InfoNCE) — arXiv:1807.03748
  • Dosovitskiy et al., An Image is Worth 16x16 Words (ViT) — arXiv:2010.11929