Reinforcement Learning

Reinforcement learning trains an agent to act in an environment by trial and error, using only a scalar reward signal. There are no labels; the agent must discover which actions lead to good outcomes and credit them correctly. The core TensorFlow pattern is small: a network that predicts action values, updated from sampled experience.

The idea

An agent interacts in a loop:

1. Observe state s.
2. Choose action a, often epsilon-greedy over predicted values.
3. Receive reward r and next state s'.
4. Store (s, a, r, s') and learn from a batch of such transitions.

Q-learning learns Q(s, a), the expected discounted return after taking a in s. The Bellman target is:

The network's loss is (Q(s, a) - y)². Two stabilizers make this work beyond toy problems:

• Replay buffer — sample random past transitions so consecutive updates are not correlated.
• Target network — compute y with a slowly updated copy, so the target does not chase the online network's own output.

Worked example

CartPole-v1 has a 4-value observation (position, velocity, angle, angular velocity) and two actions.

• Network: Dense(24, relu) → Dense(24, relu) → Dense(2).
• Discount γ = 0.99, buffer of 50,000 transitions, batch 64.
• Epsilon decays from 1.0 to 0.05.
• The target network is copied from the online network every 1,000 steps.

Each episode ends when the pole falls or at 500 steps. The 2-unit output layer is deliberate: one Q-value per discrete action, read with tf.argmax.

In code

import tensorflow as tf

q_net = tf.keras.Sequential([
    tf.keras.layers.Dense(24, activation="relu", input_shape=(4,)),
    tf.keras.layers.Dense(24, activation="relu"),
    tf.keras.layers.Dense(2),
])
target = tf.keras.models.clone_model(q_net)
target.set_weights(q_net.get_weights())
opt = tf.keras.optimizers.Adam(1e-3)

def train_step(states, actions, rewards, next_states, dones, gamma=0.99):
    y = rewards + gamma * (1.0 - dones) * tf.reduce_max(target(next_states), axis=1)
    with tf.GradientTape() as tape:
        q = tf.gather_nd(q_net(states),
                         tf.stack([tf.range(tf.shape(actions)[0]), actions], axis=1))
        loss = tf.reduce_mean(tf.square(y - q))
    opt.apply_gradients(zip(tape.gradient(loss, q_net.trainable_variables),
                            q_net.trainable_variables))
    return loss

tf.stop_gradient around y is what keeps the loss from differentiating through the target.

Check yourself

1. Why does the target network prevent divergence that a single network suffers?
2. What role does (1.0 - dones) play in the Bellman target?
3. Why must transitions be sampled randomly from the replay buffer?

Key takeaways

• RL learns a value function from reward; the label is the Bellman target, not a human tag.
• Replay and a target network exist to break the correlation between updates.
• One output unit per discrete action turns control into an ordinary regression with argmax selection.