Entropy & Information Theory

Entropy measures how much uncertainty a probability distribution carries, in bits or nats. It is the average surprise of a sample: common outcomes are cheap, rare outcomes are expensive. Cross-entropy and perplexity, the two numbers you watch during language model training, are both built from it.

The idea

For a discrete random variable with probabilities $p(x)$:

The quantity $-\log p(x)$ is the surprise (also called self-information) of outcome $x$. Entropy is its expectation over the distribution. A certain event has $p = 1$, surprise $0$, and contributes nothing.

Facts worth memorizing:

• Base sets the unit. $\log_2$ gives bits; $\ln$ gives nats. One nat equals $\log_2 e \approx 1.4427$ bits.
• Bounds. $H \geq 0$, with equality for a deterministic outcome. Among $k$ outcomes, $H$ is maximized by the uniform distribution at $\log k$.
• Cross-entropy $H(p, q) = -\sum_x p(x) \log q(x)$ scores a model $q$ against the true $p$. It satisfies $H(p, q) \geq H(p)$, with equality only when $q = p$.
• Perplexity is $2^{H(p,q)}$ when using bits — an effective number of equally likely choices.

Worked example

A biased coin with $p(\text{heads}) = 0.9$ and $p(\text{tails}) = 0.1$:

A fair coin gives exactly 1 bit, and a four-sided uniform die gives $\log_2 4 = 2$ bits.

Now score a model that assigns probability $q = 0.1$ to the observed class in 4-way classification with one-hot targets. Cross-entropy is $-\ln 0.1 = 2.3026$ nats, or $2.3026 \times 1.4427 = 3.3219$ bits. Since the true label is deterministic here, $H(p) = 0$ and all of the cross-entropy is error — that is the loss a classifier pushes down.

In code

import numpy as np

p = np.array([0.9, 0.1])
print(-(p * np.log2(p)).sum())      # 0.4690 bits
print(-(p * np.log(p)).sum())       # 0.3251 nats

q = np.array([0.1, 0.9])
print(-(p * np.log(q)).sum())       # 2.0829 nats cross-entropy
print((p * np.log(p / q)).sum())    # 2.0829 - 0.3251 = 1.7578 nats KL

Cross-entropy is -p log q summed; entropy is the special case q = p.

Check yourself

1. What is the entropy of a fair 8-sided die, and in what units?
2. Why is cross-entropy never less than entropy, and when are they equal?
3. A language model reports 10 bits per token of cross-entropy. What is its perplexity?

Key takeaways

• Entropy is expected surprise; it is 0 for certainty and $\log k$ for a uniform $k$-way choice.
• Cross-entropy measures a model against the truth and is minimized only by the true distribution.
• Perplexity is exponentiated cross-entropy, an interpretable effective vocabulary size.