Moving Averages (EMA)

A moving average smooths a noisy sequence by mixing in past values. The exponential moving average (EMA) gives recent points more weight, and it updates with one multiply and one add. Optimizer state, batch-norm statistics, and target networks all run on this recursion.

The idea

The simple moving average takes the mean of the last $k$ values and weights them all equally. The EMA replaces the window with a decay:

with $\beta \in [0, 1)$. Unrolling shows why it is exponential: $s_t = (1-\beta)\sum_{j \geq 0} \beta^j x_{t-j}$. The weight of a value $j$ steps back is $(1-\beta)\beta^j$, a geometric decay.

Two settings to reason about:

• Effective window is about $1/(1-\beta)$ steps. $\beta = 0.9$ averages over roughly 10 samples; $\beta = 0.99$ over roughly 100. Larger $\beta$ is smoother and slower to react.
• Bias correction removes the start-up lag. Initializing $s_0 = 0$ makes early values too small by a factor of $1-\beta^t$, so the corrected estimate is $\hat{s}_t = s_t / (1-\beta^t)$. Adam applies this to both of its moment estimates.

Because the recursion is linear, it converges to the constant level of any stationary input: if $x_t = c$ forever, $s_t \to c$.

Worked example

Let $x = (1, 2, 3)$ with $\beta = 0.5$ and $s_0 = x_0 = 1$:

• $s_1 = 0.5(1) + 0.5(2) = 1.5$
• $s_2 = 0.5(1.5) + 0.5(3) = 2.25$

Checking the weights directly: $x_2$ enters with weight $1-\beta = 0.5$, $x_1$ with $\beta(1-\beta) = 0.25$, and $x_0$ with $\beta^2 = 0.25$. Weighted sum: $0.5(3) + 0.25(2) + 0.25(1) = 1.5 + 0.5 + 0.25 = 2.25$. The weights sum to 1, as they must.

The effective window is $1/(1-0.5) = 2$ steps. With $\beta = 0.9$ the same three points would be dominated by the initial value, because the window is 10 steps long.

For bias correction, start from $s_0 = 0$. The first raw estimate is $s_1 = 0.5(1) = 0.5$, exactly half of $x_1$, but dividing by $1 - \beta^1 = 0.5$ recovers $\hat{s}_1 = 1$. As $t$ grows, $1 - \beta^t \to 1$ and the correction fades away.

In code

import numpy as np

x = np.array([1.0, 2.0, 3.0, 4.0, 3.0, 2.0])

def ema(xs, beta):
    s, out = 0.0, []
    for t, xt in enumerate(xs):
        s = beta * s + (1 - beta) * xt
        out.append(s / (1 - beta**(t + 1)))    # bias-corrected
    return np.round(out, 4)

print(ema(x, 0.5))   # [1.     1.6667 2.4286 3.2667 3.129  2.5556]
print(ema(x, 0.9))   # [1.     1.5263 2.0701 2.6313 2.7213 2.5674]

The $\beta = 0.9$ curve lags on the way up and is smoother at the peak. That lag is the price of variance reduction.

Check yourself

1. Why is the new value multiplied by $(1-\beta)$ rather than added directly?
2. How many past samples does $\beta = 0.95$ effectively average over?
3. Why does Adam correct its moment estimates for bias early in training?

Key takeaways

• EMA is a geometric-weighted average: $s_t = \beta s_{t-1} + (1-\beta)x_t$.
• $1/(1-\beta)$ is the effective window; bias correction fixes the cold start.
• It is the standard way to track a running statistic cheaply in one pass.