Adam: A Method for Stochastic Optimization

Stochastic gradients are noisy, scaled differently across parameters, and change as training progresses. A single global SGD step size is a blunt instrument: too large and some coordinates explode; too small and sparse features crawl. Adam combines two successful ideas — momentum (first moment) and RMSProp-style scaling (second moment) — into one update with bias correction for the fact that exponential averages start at zero.

The punchline: per-parameter adaptive rates with defaults that “just work” across many deep-learning problems, which is why Adam became a default optimizer for a decade of training recipes.

What this paper explains

Let $g_t$ be the minibatch gradient at step $t$. Adam maintains:

with default $\beta_1 = 0.9$, $\beta_2 = 0.999$. Because $m_0 = v_0 = 0$, early estimates are biased toward zero. Bias-corrected moments:

Parameter update (learning rate $\alpha$):

with $\varepsilon$ a small constant (paper default $10^{-8}$).

Prior limits

• SGD — simple, but learning-rate schedules and gradient scaling are fragile.
• Momentum / Nesterov — accelerate in consistent directions but still one global scale.
• AdaGrad — accumulates squared gradients forever; effective rates can shrink too aggressively.
• RMSProp — uses an exponential second moment but lacks Adam’s first-moment + bias-correction packaging.

The mechanism

First moment $m_t$ smooths the gradient direction (momentum-like). Second moment $v_t$ estimates a per-coordinate curvature/noise scale; dividing by $\sqrt{v_t}$ down-weights steps where gradients have historically been large. Bias correction matters most while $t$ is small — especially for $\beta_2$ close to 1, where $v_t$ warms up slowly.

Adam is invariant to diagonal rescaling of the gradients in a useful practical sense: if you multiply a coordinate’s gradients by a constant, the ratio $\hat{m}/\sqrt{\hat{v}}$ adjusts accordingly.

Algorithm / figure walkthrough

1. Initialize $\theta$, $m=0$, $v=0$, $t=0$.
2. Sample minibatch; compute $g_t = \nabla_\theta f_t(\theta_{t-1})$.
3. Update $m_t$, $v_t$ with $\beta_1$, $\beta_2$.
4. Form $\hat{m}_t$, $\hat{v}_t$.
5. Apply the scaled step; optionally add weight decay outside this paper’s original algorithm (AdamW is later work).

The opening visual shows dual EMA meters feeding a single adaptive step.

What to notice when reading

• Bias correction is not optional trivia — it is why early steps are well-scaled with $\beta_2=0.999$.
• Adam estimates uncentered second moments ($g^{2}$), not $(g-m)^{2}$.
• The paper’s theory section sketches regret-style arguments under standard online-convex assumptions; practice on deep nets is broader than the theorems.

Results and evidence

Kingma & Ba demonstrate competitive or better training curves vs SGD variants, AdaGrad, and RMSProp on logistic regression, multi-layer nets, and convolutional models in the paper’s experiments. They emphasize robustness to hyperparameter choices with the quoted defaults. Later literature debated generalization vs SGD on some vision tasks — that debate is downstream; this paper’s claim is efficient, robust stochastic optimization with adaptive rates.

Limitations

• Adaptive methods can underperform carefully tuned SGD on some generalization benchmarks (later empirical literature).
• Original Adam interacts awkwardly with L2 regularization → AdamW decouples weight decay.
• Second-moment EMA can be slow to react when gradient noise statistics shift abruptly.
• Memory: stores $m$ and $v$ (two extra buffers per parameter) — relevant for sharded optimizers like ZeRO.

How to read the paper

1. Abstract + §1–2 — motivation vs AdaGrad/RMSProp.
2. §2 — algorithm box (the whole method fits on one page).
3. §2.1 — bias correction derivation.
4. §4 — experiments.
5. §3 — convergence discussion (read with calibrated expectations).

Knowledge check

Keep reading

1. Original paper (arXiv:1412.6980)
2. Related Fanout Daily: ZeRO (optimizer-state memory), modern training stacks.

Sources

• Kingma & Ba, Adam: A Method for Stochastic Optimization, ICLR 2015 — arXiv:1412.6980