Layer Normalization

Deep nets train faster when the summed inputs to each nonlinearity sit in a stable range. Batch normalization (Ioffe & Szegedy) does that by estimating a mean and variance for each neuron across the mini-batch. It works brilliantly in many feed-forward ConvNets — and then trips over two walls: the estimate depends on batch size, and it is awkward inside recurrent nets whose “batch” axis mixes time steps poorly.

Layer normalization flips the axis. For one training case, take the $H$ summed inputs into a layer, compute a shared $\mu$ and $\sigma$ over those units, re-center / re-scale, then apply a learned per-unit gain $g_i$ and bias $b_i$ before the nonlinearity. Train and test use the same formula. For RNNs, recompute those stats at each time step.

The abstract’s punchline: stabilize hidden-state dynamics in recurrent nets and substantially reduce training time versus prior techniques on the paper’s suite (order-embeddings, reading comprehension LSTMs, skip-thoughts, DRAW, handwriting, MNIST MLPs) — with an honest caveat that ConvNets still preferred BatchNorm in their preliminary trials.

What this paper explains

Ba, Kiros, and Hinton give a small, sharp alternative to BatchNorm:

1. Name BatchNorm’s limits. Mini-batch statistics couple examples; train-time $\mu,\sigma$ differ from test-time running averages; RNNs do not have a clean “batch of independent cases” at each step.
2. Normalize within a layer. For summed inputs $a^l_1,\ldots,a^l_H$ on one case, set

1. Keep an affine after the normalize. Like BatchNorm, each unit gets its own gain and bias so the representation can still stretch and shift after re-centering.
2. Specialize to RNNs. At time $t$, normalize the pre-activations with $\mu^t,\sigma^t$ from that step’s $H$ units — no dependence on other sequences in the batch.
3. Analyze invariances. Show how LayerNorm’s invariance to global weight/data rescaling differs from BatchNorm and weight normalization (their §5), then validate on six tasks (§6).

This is not a new optimizer and not a regularization trick like Dropout. It is a reparameterization of the pre-nonlinearity activations that changes the geometry of learning.

Prior limits

• Unnormalized deep nets suffer from shifting activation distributions as parameters move — the “covariate shift” motivation BatchNorm popularized.
• BatchNorm estimates $\mu_i,\sigma_i$ per neuron over the batch. Small batches make noisy estimates; RNNs / online decoding want a definition that does not lean on other examples.
• Train ≠ test under BatchNorm. Training uses batch moments; inference typically uses running averages — an extra moving-target surface.
• Recurrent batch normalization existed as a fix attempt; the authors still find LayerNorm simpler and, in their reading-comprehension setup, less brittle to gain initialization.

The mechanism

From BatchNorm’s axis to LayerNorm’s axis

A standard feed-forward step (paper Eq. 1) is

BatchNorm (paper Eq. 2) replaces $a^{l}_{i}$ with a version normalized by expectations over the data distribution / mini-batch for that unit $i$.

LayerNorm instead pools across units $i=1..H$ for a fixed example (paper Eq. 3 above). Every hidden unit in the layer shares the same $\mu^{l}$ and $\sigma^{l}$. The normalized, affine-adjusted pre-activation (feed-forward form in the analysis section) looks like

with the understanding that under pure LayerNorm the $\mu,\sigma$ are the shared layer statistics (the $i$ subscripts on $\mu,\sigma$ collapse). In the common modern shorthand (and the paper’s supplementary $LN$ helper),

Why RNNs like this

At time $t$, the recurrent layer re-centers and re-scales using only $a^{t}_{1..H}$ (paper Eq. 4):

Normalization terms depend on the current summed inputs, not on other sequences or on a running average store. Sequence length at test can exceed anything seen in training without breaking a BatchNorm population-stat contract.

Invariance intuition (short)

§5 shows LayerNorm is invariant to rescaling the entire weight matrix and to rescaling a single training case, while BatchNorm’s invariances sit on a different axis (per-unit batch moments). You do not need the Fisher-geometry derivations to use the layer — but they explain why the gain $g$ behaves differently than a raw weight scale.

Algorithm / figure walkthrough

1. Receive a vector of summed inputs $\mathbf{a}\in\mathbb{R}^{H}$ for one example (or one time step).
2. Compute scalar $\mu=\mathrm{mean}(\mathbf{a})$ and $\sigma=\mathrm{std}(\mathbf{a})$ (paper uses the $1/H$ average inside the square root; frameworks often add $\varepsilon$ for stability).
3. Form $\hat{a}_i=(a_i-\mu)/\sigma$.
4. Apply learned $g_i,b_i$: $g_i\hat{a}_i+b_i$.
5. Apply nonlinearity $f$.
6. For RNNs / GRUs / LSTMs, apply the same pattern to the paper’s chosen pre-activations (supplement gives explicit GRU/LSTM expansions).
7. Default init in their experiments: adaptive gains to $1$ (they note sensitivity differs from recurrent BatchNorm’s scale init).

The opening visual is the same arc: ENTER with batch-tied BatchNorm / RNN pain; RUN with per-example feature-axis $\mu,\sigma$ plus gain/bias; LEAVE with train=test math and steadier recurrent dynamics.

What to notice when reading

• Axis is the idea. If you remember only one sentence, remember “across features, not across the batch.”
• Gain and bias are not optional decoration. Without them the layer could only emit zero-mean unit-variance pre-activations; $g_i,b_i$ restore expressive scale and shift.
• RNN results are the center of gravity. Four of six experiment sections are sequence models; the MNIST MLP probes batch-size robustness; ConvNets are explicitly preliminary and mixed.
• They do not claim ConvNet supremacy. §6.7: LayerNorm beat an unnormalized baseline but BatchNorm still looked better in their ConvNet trial — “further research is needed.”
• Modern Transformers. Pre-/post-LN Transformer blocks are descendants of this axis choice; the 2016 paper’s empirical home was RNNs first.

Results and evidence

Numbers and qualitative claims below are from the paper (arXiv:1607.06450) — not re-measured here.

Read the figures as training-dynamics evidence, not as a 2026 leaderboard.

Limitations

• ConvNet story incomplete in-paper. Authors flag that BatchNorm remained preferable in their convolutional experiments.
• $\varepsilon$ and framework details. Production LayerNorm adds a small $\varepsilon$ under the square root; match your library’s definition when reproducing.
• Where you place LN matters later. Transformer pre-LN vs post-LN, and Megatron-era notes about LayerNorm rearrangement in BERT-like stacks, are later systems/modeling choices — not settled in this 2016 paper.
• Not a free lunch on width. Very small $H$ makes $\mu,\sigma$ noisy; the method assumes a layer with enough units to estimate moments.
• Cousin methods exist. RMSNorm drops mean-centering; GroupNorm slices channels — useful comparisons, different papers.

How to read the paper

1. Abstract + §3 — axis swap and Eq. 3–4.
2. §3.1 — RNN formulation.
3. §5.1 — invariance table/intuition (skim Fisher §5.2 unless you want the geometry).
4. §6.1–6.2 — clearest empirical plots for sequence models.
5. §6.7 + conclusion — honest ConvNet caveat.
6. Supplementary — exact $LN(\cdot)$ applications to GRU/LSTM variants.

Knowledge check

Keep reading / Sources

• Paper: arXiv:1607.06450 — Layer Normalization (Ba, Kiros, Hinton)
• Related Daily: Batch Normalization, Megatron-LM (systems note on LN placement), Dropout
• Next cousins to study: RMSNorm (arXiv:1910.07467), Group Normalization (arXiv:1803.08494)