Tensor Broadcasting

Broadcasting is the rule that lets NumPy and PyTorch combine tensors of different shapes without making a copy. It is what makes adding one bias vector to a whole batch of activations a single line of code, and it is a common source of silent shape bugs.

The idea

Align the two shapes from the right. Two dimensions are compatible when they are equal, when one of them is 1, or when one is missing (treated as 1). Each size-1 dimension is virtually stretched to match; the result takes the larger size in every position. No memory is allocated for the stretched view, which is why it is fast.

The last row fails because 3 and 2 are neither equal nor 1. Broadcasting never guesses how to reshape your data; it only stretches dimensions that are already length 1.

Worked example

Let x be a batch of three rows with four features:

x = [[ 0,  1,  2,  3],
     [ 4,  5,  6,  7],
     [ 8,  9, 10, 11]]

Adding the bias b = [10, 20, 30, 40], which has shape (4,), aligns b with the feature axis and adds it to every row. Row 0 becomes [10, 21, 32, 43].

Combining a column (3, 1) with a row (1, 4) produces a (3, 4) grid whose entry (i, j) is col[i] + row[j]. For col = [[1], [2], [3]] and row = [[10, 20, 30, 40]], the last row is [13, 23, 33, 43].

In code

import torch

x = torch.arange(12, dtype=torch.float32).reshape(3, 4)
b = torch.tensor([10.0, 20.0, 30.0, 40.0])
y = x + b
print(y.shape)   # torch.Size([3, 4])
print(y[0])      # tensor([10., 21., 32., 43.])

col = torch.tensor([[1.0], [2.0], [3.0]])       # (3, 1)
row = torch.tensor([[10.0, 20.0, 30.0, 40.0]])  # (1, 4)
print((col + row).shape)  # torch.Size([3, 4])

try:
    x + torch.zeros(2, 4)
except RuntimeError as e:
    print("shape mismatch:", str(e).splitlines()[0])

Check yourself

1. What shape results from broadcasting (5, 1, 3) with (4, 3), and why?
2. Why does (3, 4) + (2, 4) raise an error while (3, 4) + (3, 1) succeeds?
3. In a linear layer, why does a bias of shape (out,) work against a batch of activations shaped (batch, out)?

Key takeaways

• Broadcasting stretches size-1 dimensions and pairs matching ones, reading shapes from the right.
• The result takes the larger size in each aligned position; non-matching, non-1 sizes fail.
• It powers bias adds and outer products without copying data, but it hides mistakes silently.