KL Divergence

KL divergence measures how much extra cost you pay when you encode data from one distribution using a model of another. It is the difference between cross-entropy and entropy, it is always non-negative, and it is not symmetric — facts that explain why it shows up in VAEs, distillation, and RLHF.

The idea

For discrete distributions $p$ and $q$:

The rewriting as a difference is the useful part for training. Since $H(p)$ is a constant of the data, minimizing cross-entropy is exactly minimizing $D_{\mathrm{KL}}(p\|q)$.

Properties to keep straight:

• Non-negative. Gibbs' inequality gives $D_{\mathrm{KL}} \geq 0$, with equality if and only if $p = q$ almost everywhere.
• Asymmetric. $D_{\mathrm{KL}}(p\|q) \neq D_{\mathrm{KL}}(q\|p)$ in general, so it is a divergence, not a distance. It also violates the triangle inequality.
• Support. If $p(x) > 0$ but $q(x) = 0$, the term is infinite. A model that assigns zero probability to a possible event is penalized without bound.
• Direction matters. Minimizing $D_{\mathrm{KL}}(p\|q)$ is mass-covering: $q$ spreads out to avoid the infinite penalty on $p$'s mass. Minimizing $D_{\mathrm{KL}}(q\|p)$ is mode-seeking: $q$ can collapse onto one mode and ignore the rest.

Worked example

Let $p = (0.5, 0.5)$ and $q = (0.9, 0.1)$. In nats,

The reverse direction:

Both are positive and they differ by about $0.14$ nats. That gap is the asymmetry, and it is small here only because the two distributions are close. If $q$ were $(1, 0)$ while $p$ kept mass on both outcomes, $D_{\mathrm{KL}}(p\|q)$ would diverge to infinity while $D_{\mathrm{KL}}(q\|p)$ stayed finite.

In code

import numpy as np

p = np.array([0.5, 0.5])
q = np.array([0.9, 0.1])

print((p * np.log(p / q)).sum())   # 0.5108  -> D_KL(p || q)
print((q * np.log(q / p)).sum())   # 0.3681  -> D_KL(q || p)

Note the argument order: the second array sits inside the $\log$, so (p * log(p / q)) is $D_{\mathrm{KL}}(p\|q)$. Swapping the two arrays silently optimizes the wrong objective.

Check yourself

1. Why is $D_{\mathrm{KL}}(p\|q)$ not considered a distance?
2. For which pair of distributions is the divergence zero, and what does that mean about the model?
3. Explain why minimizing cross-entropy and minimizing KL divergence are the same optimization problem.

Key takeaways

• KL divergence is the extra coding cost of using $q$ when $p$ is true: $H(p,q) - H(p)$.
• It is non-negative, asymmetric, and infinite when $q$ rules out an event that $p$ allows.
• Forward KL covers mass; reverse KL seeks modes; training minimizes cross-entropy, which differs from KL only by a constant.