Vectors

A vector is an ordered list of numbers that you can add, scale, and compare. Every activation, every row of a weight matrix, and every token embedding in a transformer is a vector, so the basic operations below show up on every forward pass. The two you will use most are the dot product and the norm.

The idea

A vector in $\mathbb{R}^n$ is written $a = (a_1, a_2, \dots, a_n)$. Addition is element-wise, and scaling multiplies every entry by the same number. Those two operations are the whole of linear algebra's "linear" part.

Three quantities carry most of the meaning:

• Dot product — $a \cdot b = \sum_i a_i b_i = \|a\|\|b\|\cos\theta$. It measures alignment. Positive means the vectors point roughly the same way, zero means orthogonal, negative means opposed.
• Norm — $\|a\|_2 = \sqrt{\sum_i a_i^2}$ is the Euclidean length. The $\ell_1$ norm $\sum_i |a_i|$ and the $\ell_\infty$ norm $\max_i |a_i|$ appear in regularization and clipping.
• Cosine similarity — $\cos\theta = \dfrac{a \cdot b}{\|a\|\|b\|}$. It ignores magnitude and lives in $[-1, 1]$, which is why embedding search uses it: a long document and a short query can still be "similar".

Worked example

Let $a = (3, 4)$ and $b = (1, 2)$.

• $a \cdot b = 3(1) + 4(2) = 11$
• $\|a\| = \sqrt{9 + 16} = 5$, and $\|b\| = \sqrt{5} \approx 2.236$
• $\cos\theta = 11 / (5 \times 2.236) = 11 / 11.180 \approx 0.984$, so $\theta \approx 10.3^\circ$

Now take $c = (1, 0)$ and $d = (0, 1)$. Their dot product is $0$, so they are orthogonal — uncorrelated directions. The unit vector pointing along $a$ is $a / \|a\| = (0.6, 0.8)$, obtained by dividing by the norm.

In code

import numpy as np

a = np.array([3.0, 4.0])
b = np.array([1.0, 2.0])

print(a @ b)                      # 11.0
print(np.linalg.norm(a))          # 5.0
print((a @ b) / (np.linalg.norm(a) * np.linalg.norm(b)))  # 0.9838...
print(a / np.linalg.norm(a))      # [0.6 0.8]

@ is the dot product for 1-D arrays. The same symbol becomes matrix multiplication for 2-D arrays, which is the subject of the next lesson.

Check yourself

1. If $a \cdot b = 0$, what geometric relationship holds, and what does that imply about information shared between the two feature directions?
2. Compute $\|(6, 8)\|_2$ and the unit vector in that direction.
3. Why does cosine similarity tolerate different vector magnitudes while Euclidean distance does not?

Key takeaways

• A vector is a point and a direction; addition and scaling act entry by entry.
• The dot product measures alignment, the norm measures length.
• Cosine similarity is normalized dot product — the standard comparison for embeddings.