Generative Adversarial Networks

Before 2014, many deep generative models paid a heavy systems tax: Markov chain Monte Carlo mixing, variational bounds, or a second network trained just to approximate the posterior. Sampling could be slow. Likelihoods could be awkward. The practical question was whether you could learn a generator that draws sharp samples with ordinary backpropagation.

Goodfellow et al. answer with a game. A generator $G$ maps noise $z \sim p_z$ to a sample $x = G(z)$. A discriminator $D$ outputs the probability that its input came from the real data distribution $p_{\mathrm{data}}$ rather than from $G$. Train $D$ to be a good detective; train $G$ to fool $D$. The result is a minimax two-player game whose unique solution (in function space) is $p_g = p_{\mathrm{data}}$ with $D(x) = \tfrac{1}{2}$ everywhere.

What this paper explains

Two multilayer perceptrons define the players:

• $G(z; \theta_g)$ — maps latent noise to the data space.
• $D(x; \theta_d)$ — outputs a scalar in $(0,1)$: “probability this $x$ is real.”

The value function of the game is

$D$ climbs $V$; $G$ descends it (or, in the practical non-saturating trick below, climbs $\log D(G(z))$ instead). No Markov chains at train time. No unrolled inference network. Generation is a single forward pass through $G$.

Prior limits

• Explicit density / Boltzmann machines. Often require MCMC; mixing can be slow and hard to diagnose.
• Variational / autoencoding families. Tractable bounds, but the paper frames adversarial training as an alternate route that avoids approximate inference networks during generation.
• Score matching / noise-contrastive cousins. Related “tell signal from noise” spirit exists, but GAN’s move is to learn the generator and the adversary jointly so the decision surface co-evolves with $p_g$.
• Wish list from the abstract. Backprop-only training; no chains while sampling; few restrictions on the model beyond differentiability of $G$ and $D$.

The mechanism

Discriminator step. Sample a minibatch of real $x$ and fake $G(z)$. Ascend the stochastic gradient of

so $D$ rises on real data and falls on fakes.

Generator step. With $D$ held fixed for the update, push $G$ so fakes look real. The literal game uses $\log(1 - D(G(z)))$, but early in training $D$ can reject fakes confidently and that objective saturates. The paper’s practical fix: maximize $\log D(G(z))$ instead — same fixed point, stronger early gradients.

Theory sketch (non-parametric). For fixed $G$, the optimal discriminator is

Plugging $D^*_G$ into the training criterion yields a cost $C(G)$ that equals $-\log 4 + 2\,\mathrm{JSD}(p_{\mathrm{data}} \Vert p_g)$. Jensen–Shannon divergence is $\ge 0$ and zero iff the distributions match, so the global minimum is exactly $p_g = p_{\mathrm{data}}$, where $D^* = \tfrac{1}{2}$.

Algorithm 1. Alternate $k$ discriminator steps and one generator step (experiments use $k = 1$). Keep $D$ near-optimal while $G$ moves slowly — analogous in spirit to how PCD keeps a chain warm between updates, but without running a chain inside the loop.

Algorithm / figure walkthrough

1. Draw minibatch noise $\{z^{(1)},\ldots,z^{(m)}\}$ and data $\{x^{(1)},\ldots,x^{(m)}\}$.
2. Update $\theta_d$ by ascending the discriminator gradient (real up, fake down).
3. Optionally repeat step 2 for $k$ inner iterations.
4. Draw a fresh noise minibatch; update $\theta_g$ so $G(z)$ looks more real under the current $D$ (prefer the non-saturating $\log D(G(z))$ objective in practice).
5. Repeat. Samples are $G(z)$ — no burn-in.

Figure 1 in the paper is the pedagogical density cartoon: $D$ rises toward the optimal ratio, $G$ mass flows toward regions $D$ still calls “data-like,” and eventually $p_g$ locks onto $p_{\mathrm{data}}$ with $D \equiv \tfrac{1}{2}$.

What to notice when reading

• The game is the method. Densities $p_g$ are implicit — defined by pushing $z$ through $G$ — which is why Parzen-window estimates appear in the experiments instead of an exact likelihood.
• Saturation of $\log(1 - D(G(z)))$ is not a footnote; it motivates the non-saturating generator loss used in essentially every later GAN recipe.
• Theoretical uniqueness holds in function space with enough capacity. MLPs introduce non-convex parameterizations; the paper treats that as an empirical bet, not a finished theory of SGD.
• Samples in their figures are claimed as fair draws, with a nearest-neighbor column to argue against pure memorization — qualitative evidence alongside Table 1.

Results and evidence

Numbers below are from the paper (arXiv:1406.2661), not independent re-runs.

Setup. Adversarial nets on MNIST, Toronto Face Database (TFD), and CIFAR-10. Generator: ReLU/sigmoid mix. Discriminator: maxout + dropout. Likelihood proxy: fit a Gaussian Parzen window to generated samples; report mean log-likelihood of the test set under that kernel density (σ via validation). The authors note this estimator has high variance and struggles in high dimension — best available for models that sample but do not expose an exact density.

Table 1 (Parzen log-likelihood).

On MNIST (real-valued version in their comparison), adversarial nets lead the listed baselines. On TFD, Stacked CAE is higher; adversarial nets sit in the competitive band. Figures 2–3 show MNIST / TFD / CIFAR samples (fully connected and a conv discriminator + “deconvolutional” generator variant) without claiming superiority over every prior visual sample set — “at least competitive,” in the authors’ words.

Limitations

Compressed from the paper’s own caveats and scope:

• No exact likelihood; evaluation leans on Parzen windows and sample visuals.
• Theory assumes enough capacity and (for Proposition 2) an optimal $D$ each outer step — practical $k=1$ is a compromise.
• Non-convex MLP parameterizations admit bad critical points; mode collapse and training instability become community themes after this paper (not fully solved here).
• Early training can leave $G$ with vanishing gradients under the literal $\log(1-D)$ objective unless you switch to the non-saturating form.
• CIFAR results are qualitative in the main narrative; Table 1 covers MNIST and TFD.

How to read the paper

1. Abstract — two models, minimax, no chains, backprop-only.
2. §3 + Eq. (1) + Figure 1 — value function and the density cartoon.
3. Algorithm 1 — the $k$-step $D$ / one-step $G$ loop ($k=1$ in experiments).
4. §4 Theorems — optimal $D^*$, JSD reduction, convergence sketch.
5. §5 + Table 1 + Figures 2–3 — Parzen numbers and samples.
6. Closing discussion — where adversarial nets sit among other generative frameworks.

Knowledge check

Keep reading / Sources

• Paper: Generative Adversarial Networks (arXiv:1406.2661)
• Related Fanout Daily: Auto-Encoding Variational Bayes (likelihood-based cousin with an encoder)
• Related Fanout Daily: Denoising Diffusion Probabilistic Models (later generative paradigm)
• Related Fanout Daily: CLIP (another dual-network training story — contrastive, not adversarial)