ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Masked language modeling (MLM) made bidirectional pre-training standard — and expensive. BERT-style models pick a small subset of tokens (typically ~15%), replace them with [MASK], and train the network to recover those identities. Most of the sequence contributes context but no loss. ELECTRA’s punchline is sample efficiency: corrupt the input with plausible replacements from a small generator, then train a discriminator that labels every token as original vs replaced. The learning signal covers the whole sequence.

What this paper explains

Clark, Luong, Le, and Manning introduce replaced token detection (RTD) as an alternative pre-training objective:

1. Sample a subset of positions (still typically ~15%, like BERT).
2. Mask those positions and train a small generator $G$ with ordinary MLM to propose replacement tokens.
3. Build a corrupted sequence by inserting generator samples (if $G$ happens to emit the true token, that position stays “real”).
4. Train a discriminator $D$ with a sigmoid head on every position: was this token replaced?
5. After pre-training, throw away $G$ and fine-tune only $D$ (the ELECTRA encoder) on GLUE / SQuAD-style tasks.

The models are structured like a GAN pair, but the generator is trained with maximum likelihood, not adversarially — GANs on discrete text are hard, and their RL attempt underperformed MLE (paper Appendix F).

Prior limits

• Sparse supervision. MLM only defines loss on the masked subset. The paper’s efficiency critique: you pay a full bidirectional encode, then learn from ~15% of tokens.
• [MASK] mismatch. BERT sees artificial [MASK] tokens in pre-training that never appear at fine-tuning (XLNet avoids this differently). ELECTRA’s corrupted inputs are ordinary vocabulary tokens.
• Compute accessibility. Strong MLM systems often need large compute budgets. The authors argue compute efficiency should be a first-class metric alongside absolute scores — Figure 1 plots downstream quality vs FLOPs.
• Not “just make BERT bigger.” Scaling helps, but ELECTRA asks whether a denser objective extracts more from the same FLOPs.

The mechanism

Generator. An encoder (usually a smaller Transformer) maps the masked sequence to contextual vectors and predicts token identities at masked positions with a softmax over the vocabulary — standard MLM.

Discriminator. Another encoder maps the corrupted sequence (no [MASK] left; replacements are real wordpieces) to vectors $h_D(\mathbf{x})_t$. A linear head scores

Joint objective. Minimize generator MLM loss plus a weighted discriminator loss over all positions:

with $\lambda=50$ in their main setup (selected from $\{1,10,20,50,100\}$ in early sweeps). Gradients from $\mathcal{L}_{\mathrm{Disc}}$ are not back-propagated through the discrete samples into $G$.

Why density matters. $\mathcal{L}_{\mathrm{Disc}}$ sums over $t=1\ldots n$, not only the masked index set. Their ablation ELECTRA 15% (discriminator loss only on the originally masked 15%) closes much of the gap toward BERT — evidence that “learn from all tokens” is doing real work, not just the replace-vs-[MASK] trick alone.

Smaller generators & tied embeddings. Same-size $G$ and $D$ roughly doubles FLOPs per step vs MLM-only. They find generators about 1/4–1/2 the discriminator size work best; too-strong generators make the discrimination task overly hard. They share token/position embeddings between $G$ and $D$ (full weight tying helps little once sizes differ).

Algorithm / figure walkthrough

1. Enter — mask. Sample mask set $\mathbf{m}$ (~15% of tokens); form $\mathbf{x}^{\mathrm{masked}}$.
2. Run — generate. $G$ proposes replacements; build $\mathbf{x}^{\mathrm{corrupt}}$ by swapping masked spots for samples.
3. Leave — discriminate. $D$ labels every position original vs replaced; update both networks (MLE for $G$, BCE-style disc loss for $D$).
4. Deploy. Discard $G$; fine-tune $D$ with a thin task head (GLUE classifiers; XLNet-style span module for SQuAD in their setup).

The opening scene is the same three beats: tokens enter as a masked proposal, the generator runs to fill plausible fakes, and the discriminator leaves binary labels across the whole sequence. The slider above is a teaching dial for how much of the sequence carries loss — MLM-sparse vs RTD-dense — not a replay of Table 1 FLOPs.

What to notice when reading

• RTD is not an adversarial GAN: $G$ maximizes likelihood of true masked tokens; it is not trained to fool $D$.
• If $G$ samples the correct token, that position is labeled real — a detail they found moderately helpful.
• Weight sharing and generator size are engineering levers for FLOPs, not afterthoughts — Figure 3 in the paper.
• Efficiency ablations (Replace-MLM, All-Tokens MLM, ELECTRA 15%) isolate which piece of the recipe matters.
• Only the discriminator transfers; the generator is scaffolding.

Results and evidence

Numbers below are from the paper (arXiv:2003.10555 / ICLR 2020) — authors’ reported measurements on their setups, not timeless leaderboard truth.

Small models (Table 1, GLUE dev average). Matched ~14M-parameter, 4-day single-V100 training:

Abstract / intro claims they highlight: ELECTRA-Small has about 1/20 the parameters and 1/135 the pre-training compute of BERT-Large, yet beats a same-size BERT by ~5 GLUE points and outperforms GPT despite far less compute. Partially trained ELECTRA-Small checkpoints still score strongly (e.g. 12h → 76.0; 6h → 74.1 in Table 1).

Large models. ELECTRA-Large sized like BERT-Large: ELECTRA-400K uses roughly 1/4 the pre-training compute of RoBERTa and performs comparably to RoBERTa/XLNet; training longer (ELECTRA-1.75M) outperforms them at similar compute and, in their writeup, beats ALBERT on GLUE and sets a then-SOTA on SQuAD 2.0. Table 2/3 list full task breakdowns — cite those tables directly rather than memorizing every cell.

Compute curves. Figure 1 is the conceptual result: RTD dominates MLM across matched FLOPs budgets in their comparison.

Limitations

Compressed from the paper’s own discussion and experimental scope:

• Two networks during pre-training. Even a small $G$ adds FLOPs per step vs pure MLM; fair comparisons must count generator cost (they do in FLOPs plots).
• Not adversarial. Do not import GAN equilibrium intuitions wholesale; $G$ is an MLE proposal model.
• Encoder-only transfer. Sequence generation / decoder pre-training is out of scope here.
• Hyperparameters matter. $\lambda$, generator size, and mask rate were tuned; the widget’s “signal density” dial is pedagogy, not a claim that arbitrary mask rates match Table 1.
• Leaderboards move. Absolute GLUE/SQuAD numbers are 2020-era; learn the objective, not the frozen SOTA.

How to read the paper

1. Abstract + §1 — sparse MLM critique and the all-token discriminator pitch.
2. §2 / Figure 2 — RTD algorithm, losses, non-adversarial training.
3. §3.2 — weight sharing, generator size (Figure 3), $\lambda=50$.
4. §3.3–3.4 — Tables 1–3 small/large GLUE; SQuAD discussion.
5. Efficiency analysis — ELECTRA 15% and related ablations.
6. Related work + conclusion — vs BERT, XLNet, RoBERTa, GAN-style ideas.

Knowledge check

Keep reading

1. Original paper (arXiv:2003.10555) — Figure 2 method, Tables 1–3, efficiency ablations.
2. Related Fanout Daily: BERT, GAN.
3. Devlin et al., BERT — arXiv:1810.04805; Liu et al., RoBERTa — arXiv:1907.11692.

Sources

• Clark et al., ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, ICLR 2020 — arXiv:2003.10555 · HTML
• Fanout Daily: BERT, GANs