Layer Normalization

Why it matters

BatchNorm couples every example to the rest of the mini-batch and fights RNNs. LayerNorm computes mean and variance over the hidden units of one example so train equals test and sequence models stay stable.

What to notice

• Batch normalization’s μ and σ are estimated over the mini-batch — awkward for RNNs and different at train vs test.
• Layer normalization takes mean and variance across the H summed inputs of one layer on one training case, then applies per-unit gain and bias.
• Same computation at train and test; per-timestep stats make it a natural fit for recurrent nets.

Explore

Contrast LayerNorm with BatchNorm’s batch axis, then follow RMSNorm’s mean-free cousin and GroupNorm’s channel groups in later Fanout issues.

Explore on Fanout