Proximal Policy Optimization Algorithms

Abstract

The paper proposes a new family of policy gradient methods that alternate between sampling data through interaction with the environment and optimizing a "surrogate" objective function using stochastic gradient ascent.

• Gradient ascent is used since we want to maximize the reward (as opposed to gradient descent, which minimizes a loss).
• In RL, we try to maximize the expected cumulative reward.
• It's stochastic because you can't calculate the cumulative reward over all possible trajectories — only the sampled ones from the mini-batch.

Introduction

We want a method that is:

• Scalable — to large models and parallel implementations
• Data Efficient
• Robust — works on many problems without hyperparameter tuning

The landscape of existing approaches:

• Q-learning with function approximation fails on many simple problems and is poorly understood.
• TRPO is complicated — requires second-order optimization, conjugate gradients, and line search.
• Vanilla policy gradient methods are not data efficient or robust.

Two Approaches to Learning a Policy

1. Value-function based: Learn a value function that estimates "how good is each action in this state" and choose the action with the highest value. Only works well with discrete actions. Examples: Q-learning, DQN.
2. Policy gradient: Parametrize the policy as a neural network that takes the state as input and outputs a probability distribution over actions. Use gradient ascent to adjust parameters so the policy produces actions that lead to higher rewards. Examples: REINFORCE, PPO, TRPO.

The distinction: "Which action is the best action?" (value function) vs "For this state, what should I do?" (policy gradient). However, policy gradient methods have high variance — you might get lucky, or not.

Background: Policy Gradient Methods

Policy gradient methods work by estimating the gradient of expected reward with respect to the policy parameters, then using stochastic gradient ascent to improve.

The most commonly used gradient estimator has the form:

where $\pi_\theta$ is a stochastic policy, $\widehat{A}_t$ is the estimator of the advantage function at timestep $t$, and $\widehat{\mathbb{E}}_t$ denotes an empirical average over a finite batch of samples.

In practice, automatic differentiation software constructs an objective function whose gradient is the policy gradient estimator:

Intuition for the Method

You have a policy parametrized by a neural network ($\theta$). The policy maps states to action probabilities. You want to adjust $\theta$ so the policy collects more rewards.

The tricky part: the objective has an expectation over trajectories that depend on $\theta$ in a complex way:

How do you differentiate through this? Using the log-derivative trick:

This lets us rewrite the gradient of an expectation as an expectation we can sample:

The process becomes: run policy → collect trajectories → for each (state, action) pair compute reward × gradient → average across samples. This is the REINFORCE algorithm.

Derivation of the Policy Gradient Estimator

Setup

We have a parameterized policy $\pi_{\theta}(a_t|s_t)$ and want to maximize the expected return:

where $\tau = (s_0, a_0, s_1, a_1, \ldots)$ is a trajectory sampled under policy $\pi_{\theta}$.

Step 1: Probability of a trajectory

where $\rho(s_0)$ is the initial state distribution and $P(s_{t+1}|s_t,a_t)$ is the environment dynamics.

Step 2: Gradient of the objective

Step 3: The log-derivative trick

Using $\nabla_{\theta} P(\tau|\theta) = P(\tau|\theta) \cdot \nabla_{\theta} \log P(\tau|\theta)$:

Step 4: Simplify the log-trajectory probability

Taking the gradient w.r.t. $\theta$:

Step 5: The REINFORCE gradient

Step 6: Introduce the advantage function

Using $R(\tau)$ produces high variance. Replace it with $\widehat{A}_t = Q(s_t, a_t) - V(s_t)$ without changing the expected gradient:

Step 7: The policy gradient estimator

Estimate this expectation by sampling and averaging:

The Advantage Function

Building Blocks

Value function $V(s_t)$ — "If I'm in state $s_t$ and follow my policy from here, what's my expected total future reward?"

Action-value function $Q(s_t, a_t)$ — "If I take specific action $a_t$ in state $s_t$, then follow my policy after that?"

Note that $V$ is just $Q$ averaged over all actions weighted by the policy: $V(s_t) = \sum_{a} \pi(a|s_t) \cdot Q(s_t, a)$.

Definition

Why This Definition? — Variance Reduction

The "something" multiplying $\nabla_\theta \log \pi_\theta(a_t|s_t)$ in the policy gradient can be:

1. Total trajectory reward $R(\tau)$ — valid but high variance
2. Future reward from $t$ onward $\sum_{t' \geq t} r_{t'}$ — valid, lower variance
3. $Q(s_t, a_t)$ — same expected gradient, focuses on the action's value
4. $Q(s_t, a_t) - V(s_t)$ — same expected gradient, lowest variance

Proof that subtracting $V(s_t)$ adds zero in expectation

Therefore subtracting $V(s_t)$ adds zero in expectation but dramatically reduces variance. The advantage is the optimal baseline for variance reduction in policy gradients.

Summary

By using gradient ascent with the advantage, we push up actions that have higher-than-average returns and push down those with lower-than-average returns. The advantage acts as both a filter and a multiplier.

Generalized Advantage Estimation (GAE)

The $\widehat{A}_t$ in the PPO equation means "we don't know the true $A$, so we estimate it." PPO uses GAE, which introduces a parameter $\lambda$ that trades off bias and variance.

TD Residual

First, define the temporal difference (TD) residual at timestep $t$:

The GAE Formula

where $\gamma \in [0, 1]$ is the discount factor and $\lambda \in [0, 1]$ controls the bias-variance tradeoff.

Special Cases

When $\lambda = 0$

Just the one-step TD residual. Low variance (only depends on one step of randomness) but high bias (relies entirely on $V$ being accurate).

When $\lambda = 1$

The full Monte Carlo return minus the baseline. No bias (uses actual rewards) but high variance (depends on the entire future trajectory).

Trust Region Methods

In TRPO, the objective function ("surrogate objective") is maximized subject to a constraint on the size of the policy update.

The Fundamental Problem with Policy Gradients

The policy gradient gives us a direction to move $\theta$, but says nothing about how far to move:

The learning rate $\alpha$ is the problem. This is much worse than in supervised learning, for a reason specific to RL.

Why RL Is Different From Supervised Learning

In supervised learning, your dataset is fixed. A bad gradient step makes the loss go up, but next step you compute a gradient on the same data and recover.

In RL, your dataset is generated by your policy. A bad gradient step changes your policy, which changes what trajectories you collect, which changes the quality of your next gradient estimate.

The Step Size Dilemma

• Small $\alpha$: safe but slow — you might need millions of updates.
• Large $\alpha$: fast but dangerous — risk the death spiral above.
• Adaptive methods (Adam, RMSProp) don't solve this. They adapt step sizes per-parameter, but have no notion of "how much did the policy change?"

From Parameter Space to Distribution Space

We need a measure of distance between policies, not between parameter vectors. This is where KL divergence comes in:

The Local Approximation (Surrogate Objective)

If $\pi_\theta$ is close to $\pi_{\theta_{\text{old}}}$, we can approximate the true objective by using importance sampling:

This surrogate is a good approximation only locally. Far from $\theta_{\text{old}}$, it breaks down because the state distribution diverges and the importance sampling ratio becomes unreliable.

TRPO: Constrained Optimization

TRPO formalizes "stay close" as:

The Cost of TRPO

Solving this constrained problem requires:

1. Computing the Fisher information matrix $F$ (second-order derivative of KL w.r.t. $\theta$)
2. Using conjugate gradient to approximately solve $F^{-1} g$
3. A line search to find the largest step satisfying the KL constraint

This is doable but expensive and complex. This complexity is precisely what motivates PPO.

Clipped Surrogate Objective

Define the probability ratio:

Without a constraint, maximizing $L^{CPI}(\theta) = \widehat{\mathbb{E}}_t [r_t(\theta)\widehat{A}_t]$ could lead to excessively large policy updates. PPO modifies the objective to penalize changes that move $r_t(\theta)$ away from 1:

The motivation:

• The first term is the same as TRPO's surrogate.
• The second term clips the ratio, removing the incentive to move $r_t(\theta)$ outside $[1-\epsilon, 1+\epsilon]$.
• Where $r = 1$, both terms are equal. They diverge as $\theta$ moves away from $\theta_{\text{old}}$.
• The $\min$ handles both positive and negative advantages correctly.

Adaptive KL Penalty (Alternative)

As an alternative to clipping, PPO can use an adaptive KL penalty:

The coefficient $\beta$ is adjusted dynamically:

• If $d < d_{\text{targ}} / 1.5$: update was too conservative, decrease $\beta \leftarrow \beta / 2$
• If $d > d_{\text{targ}} \times 1.5$: update was too aggressive, increase $\beta \leftarrow \beta \times 2$

In practice, this worked worse than the clipped surrogate objective.

The Progression

Algorithm

Combined Loss

If the policy and value function share parameters (common in practice), the loss function combines the policy surrogate, a value function error term, and an entropy bonus:

Truncated GAE

The implementation runs the policy for $T$ timesteps (per trajectory segment, not per episode) and uses a truncated version of GAE:

PPO, Actor-Critic Style

Experiments

Comparison of Surrogate Objectives

• Tested clipping in log space — not much improvement.
• Used MuJoCo tasks with 1 million timesteps.
• Hyperparameters: $\epsilon$ (clip), $\beta$ (KL penalty), $d_{\text{targ}}$ (target KL).
• A 2-layer MLP with tanh activations was used. Policy and value function parameters were not shared, so the entropy bonus was not used.

Continuous Control (MuJoCo)

PPO outperformed on almost every task. To showcase performance on high-dimensional problems, the authors trained on 3D humanoid tasks: running, steering, getting up off the ground, and being pelted by cubes.

Atari

PPO achieved better results across Atari games as well. Some practitioners have used the adaptive KL variant for robotics applications.

PPO for LLM Alignment

PPO became the standard RL algorithm for aligning language models via RLHF (Reinforcement Learning from Human Feedback). The connection:

1. Policy = the language model. Given a prompt (state), it generates tokens (actions).
2. Reward model = trained from human preferences (as in the RLHF paper).
3. PPO optimization = fine-tune the LM to maximize the learned reward while staying close to the original model (via KL penalty).

This pipeline — pretrain LM, train reward model from preferences, fine-tune with PPO — powered InstructGPT (Ouyang et al., 2022) and ChatGPT. PPO's simplicity and stability made it the practical choice over TRPO for this application.

Conclusion

Proximal Policy Optimization introduces a set of policy optimization methods that use stochastic gradient ascent to perform each policy update. These methods are as stable and reliable as trust-region methods, but simpler to implement, requiring only first-order optimization. Better overall performance across both continuous control and discrete action domains.