Singular Value Decomposition (SVD)

The singular value decomposition factors any matrix into two rotations and a scaling. It tells you which directions a matrix stretches, by how much, and it gives the best low-rank approximation of that matrix for free. Compression, PCA, and LoRA all rest on it.

The idea

Every real matrix $A \in \mathbb{R}^{m \times n}$ can be written

where $U$ is $m \times m$ with orthonormal columns (left singular vectors), $V$ is $n \times n$ with orthonormal columns (right singular vectors), and $\Sigma$ is $m \times n$ with non-negative entries $\sigma_1 \geq \sigma_2 \geq \dots \geq 0$ on the diagonal.

The reading: $V^\top$ rotates the input into a basis where the action of $A$ is axis-aligned, $\Sigma$ scales along those axes, and $U$ rotates back out. Three consequences:

• Rank equals the number of nonzero singular values.
• Energy along component $i$ is $\sigma_i^2$, and the fraction retained by the first $k$ components is $\sum_{i \leq k} \sigma_i^2 / \sum_i \sigma_i^2$.
• Eckart–Young — the best rank-$k$ approximation in both the Frobenius and spectral norms is $A_k = \sum_{i \leq k} \sigma_i u_i v_i^\top$, made by truncating the decomposition. The Frobenius error is $\sqrt{\sum_{i > k} \sigma_i^2}$.

LoRA applies exactly this idea to a weight update: it assumes $\Delta W$ is approximately low rank and learns two thin factors instead of the full matrix.

Worked example

Take the symmetric matrix $A = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}$. Its singular values are $4$ and $2$, with singular directions $u_1 = (1,1)/\sqrt{2}$ and $u_2 = (1,-1)/\sqrt{2}$. Expanding term by term:

The energy split is $4^2 = 16$ and $2^2 = 4$, totaling 20, so the rank-1 term captures $16/20 = 80\%$ of the energy. The best rank-1 approximation is the first term, with Frobenius error $\sqrt{4} = 2$. A rank-1 matrix cannot represent a general $2 \times 2$ matrix, so the missing 20% is exactly the information the second direction carried.

In code

import numpy as np

A = np.array([[3.0, 1.0], [1.0, 3.0]])
U, s, Vt = np.linalg.svd(A)
print(s)                       # [4. 2.]
print(U @ np.diag(s) @ Vt)     # [[3. 1.] [1. 3.]] reconstruction

A1 = s[0] * np.outer(U[:, 0], Vt[0])
print(np.linalg.norm(A - A1))  # 2.0, best rank-1 Frobenius error

np.linalg.svd returns $V^\top$ rather than $V$, and the singular values as a 1-D array in descending order.

Check yourself

1. What does a singular value of zero imply about the matrix, and about solving $Ax = b$?
2. State the Eckart–Young result in your own words.
3. Why does truncating the smallest singular values compress an image or a weight matrix with limited loss?

Key takeaways

• SVD factors any matrix into orthogonal directions and non-negative scales.
• Rank is the count of nonzero singular values; energy is the sum of their squares.
• Truncated SVD is the provably best low-rank approximation, which is why it powers compression, PCA, and low-rank adapters.