Matrices

A matrix is a grid of numbers that acts as a linear function on vectors. When a layer says nn.Linear(768, 3072), it means a matrix with 3072 rows and 768 columns, and the forward pass is one matrix-vector product. Shapes and the multiplication rule are the whole game.

The idea

A matrix $A$ of shape $(m, n)$ maps a vector $x \in \mathbb{R}^n$ to a vector $Ax \in \mathbb{R}^m$:

Two readings of that formula are equally useful:

• Rows — entry $i$ of the output is the dot product of row $i$ of $A$ with $x$.
• Columns — the output is a weighted sum of the columns of $A$, where the weights are the entries of $x$.

Other essentials:

• Transpose $A^\top$ swaps rows and columns; a symmetric matrix satisfies $A = A^\top$.
• Identity $I$ leaves every vector unchanged.
• Multiplication $(m, k)(k, n) \to (m, n)$; the inner dimensions must agree. It is associative but not commutative.
• Rank is the number of linearly independent columns. An $(m, n)$ matrix with $m < n$ must have rank at most $m$, so it compresses information — you cannot recover $x$ from $Ax$.

Worked example

Let $A = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix}$ and $x = (1, 0, -1)$.

Row view: $(1)(1) + (2)(0) + (3)(-1) = -2$ and $(4)(1) + (5)(0) + (6)(-1) = -2$, so $Ax = (-2, -2)$.

Column view: $1 \cdot (1, 4) + 0 \cdot (2, 5) + (-1) \cdot (3, 6) = (1, 4) - (3, 6) = (-2, -2)$. Same answer, and the column view explains why a zero in the input simply drops that column's contribution.

In code

import numpy as np

A = np.array([[1, 2, 3], [4, 5, 6]])
x = np.array([1, 0, -1])

print(A.shape)      # (2, 3)
print(A @ x)        # [-2 -2]
print(A.T.shape)    # (3, 2)
print(np.linalg.matrix_rank(A))  # 2

@ also chains: (A @ B) @ x costs the same as A @ (B @ x) mathematically, but the parenthesization changes the intermediate shapes and therefore the runtime.

Check yourself

1. What are the inner dimensions required for $AB$ to be defined, and what shape is the result?
2. Give two $2 \times 2$ matrices with $AB \neq BA$ and compute both products.
3. What does a rank-1 matrix do to an input space, and why does that matter for model capacity?

Key takeaways

• A matrix is a linear map; shapes record inputs and outputs.
• Matrix-vector multiplication is a dot product per row, or a weighted sum of columns.
• Rank counts independent directions and bounds how much information survives.