Derivatives

A derivative answers one question: if I nudge the input a tiny bit, how much does the output move? Training a network is nothing more than using that answer, millions of times, to decide which way to adjust each weight.

The idea

For a function f(x), the derivative is the limit of the average slope as the gap shrinks:

Three readings are worth keeping in your head at once:

• Geometric — the slope of the tangent line at x.
• Physical — a rate: output units per input unit.
• Optimization — the direction that increases f fastest.

At a minimum, the derivative is zero. That is why training looks for points where the gradient vanishes — and why saddle points and plateaus are the real obstacles, not just "the bottom of a bowl".

The partial derivative ∂f/∂xᵢ does the same thing while holding every other input fixed. Stack all the partials of a scalar output into a vector and you get the gradient — the subject of a later lesson.

Worked example

Let f(x) = x³ − 3x. Using the power rule, f'(x) = 3x² − 3.

• f'(0) = −3 → increasing x decreases f.
• f'(1) = 0 → a stationary point.
• f'(2) = 9 → increasing x increases f steeply.

So x = 1 and x = −1 are candidates for minima or maxima; the second derivative decides which. f''(x) = 6x, so x = 1 is a minimum and x = −1 is a maximum.

In code

import torch

x = torch.tensor(1.0, requires_grad=True)
f = x**3 - 3 * x
f.backward()
print(x.grad)  # tensor(0.)  -> stationary point at x = 1

Autograd is not approximating with a small h — it applies the chain rule symbolically to the operations you ran. The number is exact up to floating-point rounding.

Check yourself

1. Why is the derivative zero at a minimum, and why is that not sufficient to prove a minimum?
2. What does ∂f/∂xᵢ hold constant, and why does that matter for a multi-weight model?
3. If f'(x) = 0 but f''(x) = 0 too, what could be happening?

Key takeaways

• The derivative is a local rate, usable as a slope, a rate, or a direction.
• Optimization finds zeros of the gradient, not necessarily minima.
• Partial derivatives isolate one input at a time; gradients collect them.