Deep Q-Learning (DQN)

Deep Q-Learning is what happens when you replace the Q-table with a neural network and then spend most of the design effort fighting the instability that follows. The insight that made it work in 2015 was not the network — it was a replay buffer and a frozen target network. Both exist to keep the regression target from chasing the prediction.

The idea

Q-learning learns the optimal action-value function directly from transitions, whoever produced them. Bellman optimality gives the fixed point:

So training regresses the network toward a bootstrapped target

where $y$ is treated as a constant. Three ingredients keep that from diverging:

• Replay buffer. Store transitions $(s, a, r, s', done)$ and sample random minibatches. Consecutive frames in a game are nearly identical; random sampling breaks the correlation that makes plain SGD on a running target unstable.
• Target network $\bar\theta$ — a lagged copy of the weights, refreshed every few thousand steps. Without it, the prediction appears on both sides of the regression and the target moves as fast as the network chases it.
• ε-greedy exploration. Act greedily on $Q$ with probability $1-\epsilon$, randomly otherwise, annealing $\epsilon$ from 1.0 toward 0.05.

Practical extras from the original recipe: clip rewards to $[-1, 1]$ so one lucky event cannot dominate, use a Huber (smooth-L1) loss instead of squared error for outlier targets, and stack four frames because one frame is not Markov.

Because the update takes a max over next actions, DQN needs a small discrete action space. Continuous control is the domain of policy gradients and actor-critic methods.

Worked example

One transition: reward $r = 1.0$, next state non-terminal, $\gamma = 0.99$. The online network says $Q_\theta(s,a) = 1.5$ and the target network says $Q_{\bar\theta}(s', \cdot) = [0.4,\ 2.0,\ 0.1]$.

Squared error gives $\tfrac{1}{2}\delta^2 = 1.095$. Huber loss with $\beta = 1$ gives, since $|\delta| > 1$,

The gradient magnitude is now constant beyond the threshold, so one badly overestimated action cannot dominate a batch.

If the transition had been terminal, $y = 1.0$ and $\delta = -0.5$: the estimate is pushed down, because a terminal state means no future reward exists.

In code

import torch
import torch.nn.functional as F

def dqn_loss(batch, online, target, gamma=0.99):
    s, a, r, s2, done = batch
    q = online(s).gather(1, a.view(-1, 1)).squeeze(1)
    with torch.no_grad():
        q_next = target(s2).max(dim=1).values
        y = r + gamma * q_next * (1.0 - done)   # bootstrap only if not terminal
    return F.smooth_l1_loss(q, y)

The no_grad block is not an optimization — it is the definition of the target.

Check yourself

1. Why does the target use $\bar\theta$ instead of $\theta$, and what breaks when you remove the target network?
2. Why is (1.0 - done) multiplied into the bootstrap term?
3. DQN is off-policy, so the replay buffer is legal. Which part of the update would a policy-gradient method refuse to use?

Key takeaways

• DQN fits $Q^*$ by regression against a bootstrapped Bellman target.
• Replay and a target network are what stop that regression from chasing its own output.
• The greedy max keeps DQN in the discrete-action regime.