Autograd: Automatic Differentiation

Autograd records every operation applied to a tensor with requires_grad=True, then replays that graph in reverse to compute exact gradients. It is the machinery that turns a loss value into a weight update.

The idea

Mark a leaf tensor with requires_grad=True, run the forward pass, call loss.backward(), and read .grad. Autograd builds the graph dynamically as your code runs — there is no separate graph definition and no compilation step.

• loss.backward() fills .grad on every tensor that required grad and fed into loss.
• θ.grad has exactly the same shape as θ.
• Gradients accumulate by default. Call optimizer.zero_grad() (or set .grad = None) before the next backward, or updates compound.
• loss.backward() needs a scalar. For a non-scalar, pass a gradient= argument of matching shape to seed it.
• Wrap evaluation in torch.no_grad() to skip graph construction and save memory.
• .detach() returns a tensor that shares data but is cut off from the graph.

Under the hood it is the chain rule applied edge by edge in reverse topological order, which is why backward costs roughly one extra forward pass.

Worked example

Fit y = 2x + 1 from three points with one linear layer:

• Inputs x = [[0.], [1.], [2.]], targets y = [[1.], [3.], [5.]]
• Forward: pred = x @ W + b, then loss = ((pred - y) ** 2).mean(), a 0-dim tensor
• loss.backward() sets weight.grad to shape (1, 1) and bias.grad to shape (1,)
• optimizer.step() moves each parameter by -lr * grad, and optimizer.zero_grad() clears the grads for the next round

A gradient check on f(x) = x³ at x = 2 returns 12.0 exactly, matching 3x² — not a finite-difference approximation.

In code

import torch

x = torch.tensor(2.0, requires_grad=True)
(x**3).backward()
print(x.grad)                        # tensor(12.)  exact: 3x^2 at x = 2

x.grad = None                        # clear, then reuse the same leaf
(x * 4).backward()
print(x.grad)                        # tensor(4.)

w = torch.randn(1, 1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
pred = torch.tensor([[0.], [1.], [2.]]) @ w + b
loss = ((pred - torch.tensor([[1.], [3.], [5.]])) ** 2).mean()
loss.backward()
print(w.grad.shape, b.grad.shape)    # torch.Size([1, 1]) torch.Size([1])

Check yourself

1. Why does loss.backward() fail when loss is a non-scalar tensor?
2. If you forget optimizer.zero_grad(), what happens to w.grad after two backward passes?
3. What does x.detach() do, and when would you use it?

Key takeaways

• Set requires_grad=True, call backward() on a scalar loss, read .grad.
• Gradients accumulate, so clear them between optimization steps.
• no_grad() and detach() both stop tracking; the first for evaluation, the second for data handoffs.