L1 vs L2 Norms

A norm measures the size of a vector. Two of them dominate practice: L1, the sum of absolute values, and L2, the Euclidean length. They differ in how they treat many small numbers versus one large one, and that difference drives regularization, robustness, and distance in embedding space.

The idea

The family of $p$-norms is

with two members used constantly:

• L2 — $\|x\|_2 = \sqrt{\sum_i x_i^2}$, the straight-line length. It is smooth away from the origin and grows quadratically, so one large coordinate dominates the total.
• L1 — $\|x\|_1 = \sum_i |x_i|$, the "taxicab" distance along the axes. It grows linearly, and every coordinate contributes equally.
• L∞ — $\|x\|_\infty = \max_i |x_i|$, the largest single component.

The geometry explains the machine learning behavior. In two dimensions, the L2 unit ball is a circle, but the L1 unit ball is a diamond with corners exactly on the axes. Adding an L1 penalty therefore pushes solutions onto an axis, meaning some weights become exactly zero — this is sparsity, the basis of Lasso. An L2 penalty shrinks all weights smoothly toward zero without producing exact zeros. For the same reason, L2 loss is sensitive to outliers while L1 loss is not.

Worked example

For $x = (3, -4)$:

• $\|x\|_1 = 3 + 4 = 7$
• $\|x\|_2 = \sqrt{9 + 16} = 5$
• $\|x\|_\infty = 4$

Now compare two 4-dimensional error vectors: $e_1 = (1,1,1,1)$ and $e_2 = (0,0,0,4)$. Both have L1 norm $4$, but $\|e_1\|_2 = 2$ while $\|e_2\|_2 = 4$. A single error of $4$ costs twice as much as four errors of $1$ under L2, which is exactly why L1 is the robust choice.

In code

import torch

x = torch.tensor([3.0, -4.0])
print(torch.linalg.vector_norm(x, ord=1))                # tensor(7.)
print(torch.linalg.vector_norm(x, ord=2))                # tensor(5.)
print(torch.linalg.vector_norm(x, ord=float("inf")))     # tensor(4.)
print(x / torch.linalg.vector_norm(x, ord=2))            # tensor([ 0.6000, -0.8000])

Dividing by the L2 norm projects the vector onto the unit circle, the standard step before a cosine similarity.

Check yourself

1. Which of $(1,1,1,1)$ and $(0,0,0,4)$ has the larger L2 norm, and why does that matter when training on errors?
2. Why does L1 regularization tend to produce exact zeros while L2 shrinks weights smoothly?
3. Why is cosine similarity built on L2 normalization rather than L1 normalization?

Key takeaways

• L2 measures straight-line length and squares large components; L1 adds absolute values and treats every component equally.
• The corners of the L1 unit ball are what make L1 penalties sparse and L2 penalties smooth.
• L1 is robust to outliers, L2 is smooth and differentiable, and both appear everywhere in model training.