Single Neuron From Scratch

A single neuron is the smallest trainable unit in deep learning: a weighted sum of its inputs, shifted by a bias, then passed through a nonlinearity. Arrange a row of those units and you have a layer; stack layers and you have a network. Build one by hand here and the rest of the course becomes bookkeeping.

The idea

The forward pass has two steps:

• Weights w hold one number per input — the direction the neuron is looking for. The dot product is large when the input points the same way, so w behaves like a learned template.
• Bias b shifts the threshold. Without it, every decision boundary must pass through the origin.
• Activation σ supplies the nonlinearity: ReLU(z) = max(0, z), sigmoid(z) = 1/(1+e^{-z}), tanh(z).

Only w and b are trainable. Backprop multiplies three factors for every weight:

That is the whole trick. Everything else in a training run is bookkeeping over millions of those products.

Worked example

Take x = (1.0, 2.0), w = (0.5, 0.4), b = -0.6, a ReLU activation, and target 1.0.

• z = 0.5·1.0 + 0.4·2.0 − 0.6 = 0.7
• a = ReLU(0.7) = 0.7
• L = (a − 1.0)² = 0.09

Because z > 0, the ReLU derivative is 1, so ∂L/∂z = 2(a − 1.0) = −0.6. The weight gradients are that number times each input: ∂L/∂w = (−0.6, −1.2) and ∂L/∂b = −0.6. One gradient step at lr = 0.1 gives w = (0.56, 0.52), b = −0.54, a new pre-activation of 1.06, and a loss of 0.0036 — down from 0.09.

If z is zero or negative, ReLU returns a zero derivative and the unit receives no gradient at all. It is dead, and no amount of data revives it, which is why initialization and learning rate matter so much.

In code

import torch

x = torch.tensor([1.0, 2.0])
w = torch.tensor([0.5, 0.4], requires_grad=True)
b = torch.tensor(-0.6, requires_grad=True)

loss = (torch.relu((w * x).sum() + b) - 1.0) ** 2
loss.backward()
print(round(loss.item(), 4))        # 0.09
print(w.grad, b.grad)               # tensor([-0.6000, -1.2000]) tensor(-0.6000)

with torch.no_grad():
    w -= 0.1 * w.grad
    b -= 0.1 * b.grad
print(round(((torch.relu((w * x).sum() + b) - 1.0) ** 2).item(), 4))   # 0.0036

dead = torch.tensor([0.0, 0.0], requires_grad=True)
((torch.relu((dead * x).sum()) - 1.0) ** 2).backward()
print(dead.grad)                    # tensor([0., 0.])

Check yourself

1. If z = -0.3 for the same inputs and target, what gradient reaches w, and why does learning stall there?
2. What can a neuron with b fixed at 0 never represent, and why does the bias fix that?
3. How many trainable parameters does one neuron with 768 inputs have?

Key takeaways

• A neuron computes σ(w·x + b): a dot product, a threshold shift, and a nonlinearity.
• Its gradient is three multiplied factors — upstream gradient, activation derivative, input.
• Where the activation derivative is zero, no gradient flows and no learning happens.