Similarity With Dot Product

When a model says two sentences are similar, it is usually taking a dot product of their embeddings and comparing the result to a threshold. The dot product is the smallest operation in deep learning that carries a notion of agreement, and attention scores, retrieval rankings, and recommendation scores are all built on it.

The idea

The dot product of two vectors of the same length multiplies matching coordinates and sums the results:

Read that second form as two separate signals:

• Direction — $\cos\theta$ is positive when the vectors point the same way, zero when they are perpendicular, and negative when they oppose each other.
• Magnitude — the lengths $\|a\|$ and $\|b\|$ scale the score, so a long vector can outscore a better-aligned short one.

Cosine similarity divides the magnitudes out: $\cos\theta = \frac{a \cdot b}{\|a\|\|b\|}$, which ranges from $-1$ to $1$. For unit vectors the dot product is the cosine, so normalizing first turns a magnitude-sensitive score into a pure direction score.

Unlike Euclidean distance, the dot product is not a metric: it has no triangle inequality, and it is maximized rather than minimized for identical vectors. The two are related by $\|a-b\|^2 = \|a\|^2 + \|b\|^2 - 2\,a\cdot b$.

Worked example

Take $a = (3, 4)$ and $b = (4, 3)$:

• $a \cdot b = 3\cdot4 + 4\cdot3 = 24$
• $\|a\| = \|b\| = 5$, so $\cos\theta = 24/25 = 0.96$ — nearly aligned.

Now make them orthogonal: $a = (0,1)$, $b = (1,0)$ gives $0\cdot1 + 1\cdot0 = 0$. And scale $b$: $a = (3,4)$, $b = (6,8)$ gives a dot product of $50$ but a cosine of $1.0$. Same direction, twice the length, a much larger raw score.

In code

import torch
import torch.nn.functional as F

a = torch.tensor([1.0, 2.0, 3.0])
b = torch.tensor([2.0, 4.0, 6.0])  # same direction, twice as long

print(torch.dot(a, b))                                  # tensor(28.)
print(F.cosine_similarity(a, b, dim=0))                 # tensor(1.0000)
print(F.normalize(a, dim=0) @ F.normalize(b, dim=0))    # tensor(1.0000)

The raw dot product is 28, but once both vectors are scaled to unit length the score collapses to 1.0 because the directions are identical.

Check yourself

1. Two nonzero vectors have a dot product of zero. What is the angle between them, and does that conclusion depend on their lengths?
2. Why can the dot product rank embeddings badly when vector lengths vary, and what does cosine similarity change?
3. For unit-length vectors, why is minimizing Euclidean distance the same as maximizing the dot product?

Key takeaways

• The dot product measures directional agreement, weighted by both vector magnitudes.
• Cosine similarity removes magnitude; for unit vectors it equals the dot product.
• Attention weights and similarity search are rankings over dot products of embeddings.