Gradients

A gradient collects every partial derivative of a scalar output into a single vector. It tells you which direction increases the loss fastest, and training steps in the opposite direction. Everything an optimizer does is built on this one object.

The idea

For $f(x_1, \dots, x_n)$, the gradient is

Each entry holds all other inputs fixed while measuring one local rate. Three properties matter:

• Direction — $\nabla f$ points toward the steepest ascent of $f$.
• Magnitude — $\|\nabla f\|$ is the maximum rate of increase over all unit directions.
• Geometry — $\nabla f$ is orthogonal to the level set $\{x : f(x) = c\}$ through the point.

Gradient descent uses the direction, not the magnitude, to decide where to move: $\theta \leftarrow \theta - \eta \nabla_\theta L$. The learning rate $\eta$ decides how far. Because of that first property, a gradient of zero is a stationary point — a minimum, a maximum, or a saddle. Non-smooth functions such as ReLU have a subgradient rather than a gradient at the kink, which is why frameworks pick an arbitrary value (0) there.

Worked example

Take $f(x, y) = x^2 + 2y^2$, whose partials are $\partial f/\partial x = 2x$ and $\partial f/\partial y = 4y$.

• At $(1, 1)$: $\nabla f = (2, 4)$, with magnitude $\sqrt{20} \approx 4.472$.
• With $\eta = 0.1$ the update gives $(1 - 0.2,\; 1 - 0.4) = (0.8, 0.6)$.
• Before: $f(1, 1) = 1 + 2 = 3$. After: $f(0.8, 0.6) = 0.64 + 0.72 = 1.36$.

The loss dropped, and it would keep dropping toward $(0, 0)$. The level sets here are ellipses stretched along $x$, so the gradient is steeper in $y$ — the optimizer zig-zags if $\eta$ is too large, because the update overshoots the narrow axis.

In code

import torch

x = torch.tensor([1.0, 1.0], requires_grad=True)
f = x[0]**2 + 2 * x[1]**2
f.backward()
print(x.grad)  # tensor([2., 4.])

You never write the partials by hand; autograd applies the derivative rules as it records each operation. For a loss over millions of parameters, this vector is what optimizer.step() consumes.

Check yourself

1. What is the difference between the direction the gradient points and the direction gradient descent moves?
2. Why can the gradient be zero at a point that is neither a minimum nor a maximum?
3. If the gradient magnitude is very large, what does that suggest about the learning rate?

Key takeaways

• The gradient stacks partial derivatives of a scalar function.
• It points uphill; training moves downhill by $\eta \nabla L$.
• Zero gradient means stationary, not optimal.