BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Before BERT, strong NLP transfer often meant a left-to-right language model (or a shallow mash-up of independently trained L→R and R→L models). Left-to-right LMs never see future tokens when building a representation at position $i$. For tasks like QA and NLI, that missing context hurts.

BERT’s punchline: pretrain a deep bidirectional Transformer encoder with Masked Language Modeling (MLM) so every layer can fuse left and right context, then fine-tune one model for many tasks with a thin head. Optional Next Sentence Prediction (NSP) teaches simple sentence-pair relations.

What this paper explains

BERT stacks Transformer encoder blocks (no causal mask). Two unsupervised objectives on BooksCorpus + English Wikipedia:

1. MLM — mask ~15% of WordPiece tokens; of those, 80% become [MASK], 10% a random token, 10% left unchanged (so fine-tuning sees fewer train/test mismatches). Predict the original id.
2. NSP — given sentences A and B, predict whether B follows A (50% true, 50% random).

Input packing uses [CLS] (pooled classification) and [SEP], plus segment embeddings for sentence A vs B, and position embeddings.

Prior limits

• ELMo — bidirectional LSTM, but shallow concatenation of independently trained directions; task models still feature-based or lightly tuned.
• OpenAI GPT (2018) — deep Transformer, but unidirectional; fine for generative LM, weaker for token-level fusion that needs future context.
• Feature-based transfer — freeze the encoder and train heavy task-specific nets; more parameters per task, less shared adaptation.

The mechanism

Because the prediction targets are masked tokens, the model cannot “cheat” by peeking at the token itself, yet attention is free to look everywhere else — left and right. Stacking many layers means bidirectional signals mix deeply, not only at the output.

Fine-tuning: initialize from pretrained weights, add a classification layer on [CLS] (or span heads for SQuAD), train end-to-end on labeled data with modest learning rates.

Algorithm / figure walkthrough

1. Tokenize with WordPiece; add [CLS] / [SEP] / segment + position embeddings.
2. Run $L$ encoder layers ($L=12$ base, $L=24$ large).
3. Pretrain with MLM (+ NSP in the original recipe).
4. Swap in a task head; fine-tune all parameters on the downstream dataset.

Sizes from the paper: BERT-base ($L=12$, $H=768$, $A=12$, ~110M params) and BERT-large ($L=24$, $H=1024$, $A=16$, ~340M params).

What to notice when reading

• The 80/10/10 masking trick — engineering detail that matters for fine-tuning mismatch.
• [CLS] as a learned pooled representation for classification.
• How little task-specific architecture they need once pretraining is strong.

Results and evidence

At publication, BERT set new highs on the GLUE benchmark (BERT-large around 80.5 GLUE average on the leaderboard numbers they report), SQuAD v1.1/v2.0, and several other tasks, beating prior GPT and feature-based systems by clear margins on the paper’s tables. Ablations show MLM is crucial vs left-to-right LM pretraining; NSP helps sentence-pair tasks in their experiments (later work debated NSP — read those as follow-ons).

Limitations

• [MASK] tokens appear in pretraining but not always in fine-tuning (mitigated by 80/10/10).
• NSP’s value was later questioned (RoBERTa et al.); MLM carried more of the win.
• Encoder-only: not a generative LM; seq2seq and decoder-only models took generation.
• Quadratic attention cost; long documents need later long-context methods.

How to read the paper

1. Abstract + §1 — bidirectional pretraining motivation.
2. §3 — architecture, MLM, NSP, input format.
3. §4 — fine-tuning setup.
4. §5–6 — GLUE/SQuAD results and ablations.
5. Appendix — more hyperparameters.

Knowledge check

Keep reading

1. Original paper (arXiv:1810.04805)
2. Related Fanout Daily: Attention Is All You Need, LoRA.

Sources

• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019 — arXiv:1810.04805