Agents & Environments

Reinforcement learning throws away the labeled dataset and replaces it with a loop: an agent acts, the world changes, and a number comes back telling it how well that went. Every other lesson in this module — policy gradients, DQN, PPO, GRPO — is a different way of using that number. Get the loop and its vocabulary straight first.

The idea

The formal object is a Markov decision process, a tuple $(S, A, P, R, \gamma)$. At step $t$ the agent sees a state $s_t \in S$, picks an action $a_t \in A$, and the environment responds with a reward $r_{t+1}$ and a next state $s_{t+1} \sim P(\cdot \mid s_t, a_t)$. The "Markov" part is a promise: the next state and reward depend only on the current state and action, not on earlier history.

Four words do most of the work:

• Policy $\pi(a \mid s)$ — the agent's behavior: a table, a softmax over logits, or a 671B-parameter network.
• Return $G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}$ — the objective, with discount $\gamma \in [0,1)$.
• Value $V^\pi(s) = \mathbb{E}_\pi[G_t \mid s_t = s]$ and action-value $Q^\pi(s,a)$.
• Model — transitions and rewards. With one you can plan; without one you must sample.

Two structural facts drive the algorithms. First, the discount rate trades immediate reward against future reward and keeps the sum finite in continuing tasks. Second, the data distribution depends on the policy: the agent chooses its own training set, and that set changes as the policy changes. Exploration is therefore part of the algorithm rather than a preprocessing step.

Worked example

A one-dimensional corridor with states 0 through 4, starting at 2, actions left (−1) or right (+1), clamped at the walls. Reaching state 4 yields reward 1 and ends the episode; every other step yields 0. Set $\gamma = 0.9$.

The policy "always right" produces the trajectory 2 → 3 → 4 with rewards $(0, 1)$, so $G_0 = 0 + 0.9 \cdot 1 = 0.9$, and $V(2) = 0.9$ for that policy.

The policy "always left" produces 2 → 1 → 0 → 0 …, never reaching the goal. Every reward is 0, so its return is exactly 0: no signal ever arrives to tell it that moving right was an option.

Same environment, same actions; only the return separates the two policies, and RL shifts probability mass from the second toward the first.

In code

GAMMA = 0.9

def step(s, a, n=5, goal=4):
    s2 = min(n - 1, max(0, s + a))
    return s2, (1.0 if s2 == goal else 0.0), s2 == goal

s, done, rewards = 2, False, []
while not done:
    s, r, done = step(s, +1)
    rewards.append(r)

G = sum(r * GAMMA**k for k, r in enumerate(rewards))
print(rewards, G)  # [0.0, 1.0] 0.9

Everything in this module is a way of updating something from the rewards list.

Check yourself

1. Why is the return discounted rather than a plain sum of rewards, and what does a smaller $\gamma$ make the agent prefer?
2. What makes RL's data distribution non-stationary in a way that supervised learning's is not?
3. In the corridor, what is the return of "always left" from state 2, and why does that make exploration necessary rather than optional?

Key takeaways

• An MDP is (states, actions, transitions, rewards, discount); a policy maps states to action probabilities.
• The objective is the expected discounted return, not the immediate reward.
• Because the policy generates its own data, exploration and non-stationarity are built into the problem.