Backprop in Python

Autograd is not magic and not a numerical trick. It is the chain rule applied to a recorded graph of scalar operations, which you can reproduce in about twenty lines. Writing the backward pass by hand once makes every framework's backward() call readable.

The idea

Forward mode records each operation and its inputs. Backward mode then walks the records in reverse, applying two rules at every step:

• Push forward the upstream derivative. Each node receives $\partial L / \partial(\text{node output})$ and multiplies it by the node's local derivative.
• Multiply locally, accumulate globally. For a node used in several places, gradients add at the node.

For a multiply $z = x \cdot y$ the local derivatives are $\partial z/\partial x = y$ and $\partial z/\partial y = x$, so the upstream gradient $g$ produces $x_{\text{grad}} \mathrel{+}= g y$ and $y_{\text{grad}} \mathrel{+}= g x$. For an add, both get $g$ unchanged. That is the entire mechanism.

Worked example

Take a two-node scalar network with a $\tanh$ hidden unit:

with $x = 1.5$, $w_1 = 0.8$, $b_1 = -0.2$, $w_2 = 1.3$, $t = 3.0$.

Forward: $z = 0.8(1.5) - 0.2 = 1.0$; $h = \tanh(1) = 0.7616$; $\hat{y} = 1.3(0.7616) = 0.9901$; $L = (0.9901 - 3)^2 = 4.0398$.

Backward, starting from $\partial L/\partial \hat{y} = 2(\hat{y} - t) = -4.0199$:

Then $\partial h/\partial z = 1 - h^2 = 0.4200$, so $\partial L/\partial z = -2.1947$, and $\partial L/\partial w_1 = -3.2921$, $\partial L/\partial b_1 = -2.1947$. Each parameter's gradient is the chain of the numbers after it.

In code

import numpy as np, torch

x, w1, b1, w2, t = 1.5, 0.8, -0.2, 1.3, 3.0
z = w1 * x + b1
h = np.tanh(z)
yhat = w2 * h
g = 2 * (yhat - t)          # dL/dyhat
dw2 = g * h
dz = g * w2 * (1 - h**2)
dw1, db1 = dz * x, dz
print(round(dw1, 4), round(db1, 4), round(dw2, 4))  # -3.2921 -2.1947 -3.0615

W1, B1, W2 = (torch.tensor(v, requires_grad=True) for v in (w1, b1, w2))
y = W2 * torch.tanh(W1 * x + B1)
((y - t)**2).backward()
print(W1.grad.item(), B1.grad.item(), W2.grad.item())  # matches

The manual numbers and the autograd numbers agree to four decimals. Memorizing the shapes and order of those multiplications is the difference between debugging a model and guessing.

Check yourself

1. Why does the gradient of an addition pass through unchanged while a multiplication scales it?
2. What is stored during the forward pass that the backward pass cannot recompute cheaply?
3. If $z = x + x$, what is $\partial z/\partial x$, and what does that say about accumulating gradients?

Key takeaways

• Backprop is reverse-mode chain rule over a recorded graph.
• Local rule: multiply by the node derivative; global rule: accumulate at shared nodes.
• Hand-computing a tiny backward pass is the best way to learn to read gradients.