Functions

A neural network is a function. Not "like" a function — literally a rule that takes an input and returns an output, built by composing smaller rules. Every layer is one of those rules, and training is just the search for the numbers inside them. Get the vocabulary straight here and the rest of the course stops feeling like magic.

The idea

A function f maps each input from a domain to exactly one output in a codomain. For a model, the domain is the space of possible inputs (a 768-dimensional embedding, a 224×224×3 image) and the codomain is the prediction space (a probability per class, a next-token distribution).

Two operations matter more than the rest:

• Composition — (f ∘ g)(x) = f(g(x)). Stack a linear map, then a nonlinearity, then another linear map and you have a layer, then a network. Deep learning is composition with trainable pieces.
• Invertibility — a function is invertible when f⁻¹(f(x)) = x for every x. ReLU is not invertible (all negatives map to 0), which is exactly why information can be lost in a network.

A model with parameters is a family of functions, f(x; θ). Training changes θ; the architecture fixes the shape of the family.

Worked example

Take f(x) = 2x + 1 and g(x) = x². Then:

• (f ∘ g)(3) = f(9) = 19
• (g ∘ f)(3) = g(7) = 49

Same two functions, different order, different result — composition is not commutative. This is why the order of layers in a network matters: Linear → ReLU and ReLU → Linear are different function families, and only one of them can approximate a nonlinear target.

In code

import torch

x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
linear = torch.nn.Linear(1, 1, bias=True)
relu = torch.nn.ReLU()

with torch.no_grad():
    linear.weight.fill_(2.0)
    linear.bias.fill_(1.0)

composed = relu(linear(x.unsqueeze(1))).squeeze(1)
print(composed)  # [0., 0., 1., 3., 5.]

relu ∘ linear clips every negative result to zero. Swap the order and the same weights produce a different function.

Check yourself

1. If f(x) = x² and g(x) = x + 3, what is (f ∘ g)(x) versus (g ∘ f)(x)?
2. Why does a network need a nonlinearity between linear layers?
3. What does it mean for a network to be a family of functions rather than one fixed function?

Key takeaways

• Layers are functions; a network is a composition of them.
• Composition order matters, and nonlinearity is what makes the family expressive.
• Training searches the parameters θ, not the structure f.