Hadamard Product (Element-wise Op)

The Hadamard product multiplies two arrays entry by entry. It is written $A \odot B$ and keeps the shape, unlike the dot product or matrix product. Gates, masks, and per-parameter scaling are all Hadamard products, which makes its gradient the simplest one in the course.

The idea

For arrays of the same shape, $(A \odot B)_{ij} = A_{ij} B_{ij}$. It is commutative, associative, and distributes over addition. The identity element is an array of all ones.

It is not the same operation as either cousin:

• Dot product of two vectors returns a scalar, $\sum_i a_i b_i$ — a contraction.
• Matrix product combines axes, $(m,k)(k,n) \to (m,n)$.
• Hadamard touches nothing: $n \times n$ times $n \times n$ gives $n \times n$.

Shapes still have to match, or broadcast. A scalar broadcasts against everything: $2 \odot a$ doubles every entry. A row vector broadcasts across the rows of a matrix, which is how a per-channel scale is applied.

The gradient is the reason this shows up everywhere. For $z = x \odot y$:

In reverse mode, with upstream gradient $\bar{g}$ of the same shape, the VJPs are

So each input's gradient is the other input acting as a mask. When one factor is a binary mask, the gradient flows exactly where the mask is 1 and is zeroed everywhere else — that is a gate.

Worked example

Let $a = (1, 2, 3)$ and $b = (4, -1, 0.5)$. Then $a \odot b = (4, -2, 1.5)$ element by element.

Now let $z = \sum_i a_i b_i$, a scalar built from a Hadamard product followed by a sum. Then $\partial z/\partial a = b = (4, -1, 0.5)$ and $\partial z/\partial b = a = (1, 2, 3)$. Compare with the dot product alone: the forward value is the same scalar, $4 - 2 + 1.5 = 3.5$, and the gradients are still the other vector — the sum never changed the local slopes.

In an LSTM, $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$: the forget gate $f_t$ multiplies the old cell state entry by entry, so a value near 0 erases a dimension and a value near 1 preserves it.

In code

import torch

a = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
b = torch.tensor([4.0, -1.0, 0.5])
print(a * b)          # tensor([ 4., -2.,  1.5])

(a * b).sum().backward()
print(a.grad)         # tensor([ 4., -1.,  0.5])  == b

In PyTorch * on tensors is the Hadamard product and @ is the matrix product. Mixing them up is one of the most common sources of shape bugs in model code.

Check yourself

1. How does $A \odot B$ differ from $A^\top B$ for two matrices of the same shape?
2. Why is $\partial z/\partial x = y$ for $z = x \odot y$ an element-wise statement rather than a matrix?
3. In attention, a mask of zeros is multiplied into the scores. What happens to the gradient through the masked positions?

Key takeaways

• Hadamard means element-wise, shape-preserving multiplication.
• Its gradient is the other operand, so it acts as a gate.
• Binary masks route gradients by zeroing the blocked positions.