Reinforcement Learning / Blog Posts / 2026

Proximal Policy Optimization: Reinforcement Learning with Human Feedback (with the maths behind fine-tuning LLMs)

MW
Mohammad Wasil Saleem published on October 1, 2026 • 4 min read
Table of Contents

1. Overview & Context

When we fine-tune a Large Language Model using standard next-token prediction, we teach it grammar, syntax, and basic pattern matching. But raw text generation alone doesn't guarantee that an LLM will be helpful. To truly align a model with human intent, we need a mechanism to reward desirable behavior and discourage toxic or unhelpful outputs.

With Reinforcement Learning with Human Feedback (RLHF) and its optimization algorithm, Proximal Policy Optimization (PPO), we can help align the model with human preferences with the help of Reward Modeling.

In this blog, we will explore the complete mathematical and architectural pipeline behind RLHF, walking step-by-step through preference modeling, KL divergence penalties, advantage estimation, and the clipped surrogate loss that keeps modern LLM alignment stable.

2. The InstructGPT Paradigm: From Raw Text to Human Alignment

To understand how modern alignment works, we can look back at OpenAI's foundational InstructGPT paper, which established the blueprint for aligning base models like GPT-3 with human intent. The pipeline relies on three fundamental phases:

  1. Pretraining: The model ingests massive datasets to learn grammar, facts, and next-token prediction, but it only knows how to complete text, not how to follow instructions.
  2. Supervised Fine-Tuning (SFT): The model is trained on curated conversations—human queries paired with ideal, high-quality responses written by human annotators (chat conversations). This SFT checkpoint serves as the critical starting point for the alignment phase.
  3. Reinforcement Learning from Human Feedback (RLHF): The phase where the model learns to optimize its behavior based on human values.
Note: In traditional reinforcement learning, a model learns through trial and error via a continuous feedback loop: 
                        ┌─────┐
            Prompt ───► │ LLM │───► Response ───► Human Evaluation ───► Parameter Update
                        └─────┘

While effective, keeping a human in the loop to evaluate every single model generation during training is slow, expensive, and completely unscalable.

To bypass this bottleneck, InstructGPT introduced a brilliant proxy, it replaces the human evaluator with a trained neural network, or an LLM (the Reward Model):

Prompt ⟶ LLM ⟶ Response ⟶ Reward Model ⟶ Parameter Update 
                       ┌─────┐                     ┌──────────────┐
            Prompt ───►│ LLM │───► Response ──────►│ Reward Model │───► Parameter Update
                       └─────┘                     └──────────────┘

In the sections that follow, we will unpack the machinery behind this pipeline:

Before diving into how reinforcement learning updates our model, we need to understand the data that makes it all possible, along with the standard naming conventions used in the literature.

Quick Nomenclature Note of all the different LLMs used (Total 4):

  • SFT Base (πSFT) / Reference Model: The model fine-tuned on supervised chat conversations. Generally denoted as πSFT or πref. It serves as our baseline anchor.
  • Reward Model (rψ): The scoring network trained to evaluate text quality, typically denoted as rψ (parameterized by weights ψ).
  • Policy LLM (πθ): The language model trained to generate text, typically denoted as πθ (parameterized by weights θ).
  • Critic LLM (πcritic): The language model used to estimate the value of states, denoted as πcritic; (parameterized by weights πSFT).

3. Human Preference Data

To train the reward model, we use pairwise human preference data:

Example Preference Data Structure

{
"prompt": "x: What is 2 + 2?",
"chosen": "y_w: The answer to 2 + 2 is 4.",
"rejected": "y_l: 2 + 2 is 5, but sometimes math is relative."
}

4. Reward Model

Reward Model Architecture

The Reward Model isn't built from scratch. It is initialized directly from our trained SFT checkpoint (πSFT) with some modifications:

Reward Model Input and Output

                                           ┌──────────────┐
            Prompt + Chosen (y_w) ───────► |              | ───────► Scalar Score: +2.5 (r_w)
                                           │ Reward Model │
            Prompt + Rejected (y_l) ─────► |              | ───────► Scalar Score: +0.5 (r_l)
                                           └──────────────┘

The Bradley-Terry Preference Model & Loss Function

The Reward Model is trained using the Bradley-Terry preference model, which computes the probability that the chosen response \(y_w\) is genuinely better than the rejected response \(y_l\) given a prompt \(x\):

$$P(y_w \succ y_l \vert x) = \sigma(r_w - r_l) = \frac{1}{1 + e^{-(r_w - r_l)}}$$

1. Concrete Numerical Calculation

Let's walk through how this works with actual numbers:

2. The Reward Model Loss Function

The loss function for training the Reward Model is the negative log-likelihood of the correct preference. We want to maximize this probability, which means minimizing the negative log-likelihood:

$$L_{\text{RM}}(\psi) = -\log(\sigma(r_w - r_l))$$

Calculated Loss Value:

$$L_{\text{RM}} = -\log(0.8176) \approx \mathbf{0.2014}$$

3. Derivative of the Loss

To update the Reward Model \(r_{\psi}\) via backpropagation, we calculate the derivative of the loss with respect to both scalar scores (\(r_w\) and \(r_l\)):

$$\frac{\partial L_{\text{RM}}}{\partial r_w} = \sigma(r_w - r_l) - 1 = 0.8176 - 1 = \mathbf{-0.1824}$$

$$\frac{\partial L_{\text{RM}}}{\partial r_l} = 1 - \sigma(r_w - r_l) = 1 - 0.8176 = \mathbf{+0.1824}$$

Interpretation: The negative gradient for \(r_w\) pushes its score higher, while the positive gradient for \(r_l\) pushes its score lower, aligning the Reward Model closer to human preferences.

To know more about Bradley Terry preference model, and how Reward models are trained, you can refer to the this amazing substack: Bradley-Terry Model for Reward Modeling

Penalizing Reward Models (Preventing Model Drift)

To measure how much the new policy deviates from the SFT model, we use KL divergence (Kullback-Leibler divergence).

If we only reward the LLM based on the Reward Model's score rψ, the LLM will quickly figure out how to "hack" the reward model. It might start repeating nonsense words that accidentally trigger high scores, or drifting into unnatural language.

To prevent this, penalize the LLM if it strays too far from the original SFT model, πSFT. This is done using KL divergence (Kullback-Leibler divergence), which measures how much the new policy's token, πθ, probabilities have changed compared to the reference SFT model πSFT.

The final reward R(x, y) given to the PPO algorithm is:
$$R(x, y) = r_\psi(x, y) - \beta D_{\text{KL}}(\pi_\theta(y\vert{}x) \parallel \pi_{\text{SFT}}(y\vert{}x))$$ where,
        rψ(x, y): The score from the Reward Model.
        β: A hyperparameter controlling how heavily we punish the model for drifting.
        DKL: The penalty for changing the output probabilities too drastically from the original SFT model.
Lets say, we have 3 tokens - v1, v2, and v3. For a specific token position, the probability distribution over the vocabulary is:
Reference SFT Model (πSFT): $$[0.7, 0.2, 0.1]$$
Active LLM Policy (πθ): $$[0.6, 0.3, 0.1]$$
The KL divergence is calculated as:
$$D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{SFT}}) = \sum_{i} \pi_\theta(v_i) \log \frac{\pi_\theta(v_i)}{\pi_{\text{SFT}}(v_i)}$$
$$= 0.6 \log \frac{0.6}{0.7} + 0.3 \log \frac{0.3}{0.2} + 0.1 \log \frac{0.1}{0.1}$$
$$= -0.0925 + 0.1216 + 0$$
$$D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{SFT}})= 0.0291$$
Suppose our trained reward model grades the new LLM response as r_ψ = +2.5. Therefore, the total penalized reward, R(x, y) is:
$$R(x, y) = r_{\psi} - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{SFT}})$$
$$= 2.5 - 0.0291$$ $$R(x, y)= 2.4942$$

5. Fine-tuning the Policy

In this section, we will explore how the Policy LLM is fine-tuned using Proximal Policy Optimization (PPO) to align it with human preferences.

Here, we combine everything into PPO trainging step to update LLM weights (θ).

                    ┌──────────┐                     ┌──────────────┐
        Prompt ───► │Policy LLM│───► Response ──────►│ Reward Model │───► Reward (R) ──► Update πθ with PPO
                    │  (π_θ)   │                     └──────────────┘
                    └──────────┘
In PPO/RLHF, we want to optimize a policy πθ to maximize the expected reward from a reward model rψ, while penalizing it for drifting too far from a reference model πSFT using KL divergence:
$$\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(y\vert{}x)} \left[ r(x,y) \right] - \beta D_{\text{KL}}(\pi_{\theta}(y\vert{}x) \parallel \pi_{\text{ref}}(y\vert{}x))$$

Policy LLM Architecture

The Policy LLM isn't built from scratch. It is initialized directly from our trained SFT checkpoint (πSFT) without any modifications.

PPO Clipped Surrogate Loss Function

The PPO algorithm uses a clipped surrogate loss function to update the policy parameters:

$$L_{PPO}(\theta) = - \mathbb{E}_t \left[ \min(r_t(\theta) \hat{A}_{t}, clip(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_{t}) \right]$$

where
        \(r_t(\theta)\) is the probability ratio, and
        \( \epsilon \) is the clipping parameter, and
        Ât is the estimated advantage (action-value) function.

Probability Ratio rt(θ)

The probability ratio \(r_t(\theta)\) is defined as:

$$r_t(\theta) = \frac{Probability of response under \pi_{\theta}}{Probability of response under \pi_{\theta_{SFT}}}$$ if rt > 1, the new model likes the response more than the old model did.
As the optimizer updates the weights to create a "new" model (πθ), run the exact same response through both models (πθ and πSFT) to see how much their probability outputs have shifted. If rt > 1, the new model likes the response more than the old model did.

Advantage Function Ât

PPO uses a secondary small neural network called the Critic.

Critic

  • While the Policy LLM (the Actor) πθ focuses on generating text, the Critic focuses on predicting future success. It acts as an internal baseline tracker.
  • Without a Critic, the model only knows the final score of a response, but it struggles to tell whether a specific output was uniquely brilliant or if the prompt itself was just very easy. The Critic provides that baseline expectation. (tells us whether there has been any improvement so far).

Critic Architecture

  • The Base Structure: The Critic shares the exact same underlying architecture as SFT LLM, πSFT.
  • The Modification: Just like the Reward Model, the final text-generation layer (the language modeling head that outputs vocabulary tokens) is stripped off. It is replaced with a linear regression head that outputs a single scalar value.

Inputs and Outputs

  • Input: Prompt + Response (generated so far).
  • Output: A single scalar number representing the expected final reward the model should receive from that point onward.
The Advantage Ât is given by:
$$\hat{A}_t = \text{Actual Reward Received} - \text{Expected Reward (Value Function from Critic)}$$

Clipping Parameter ε

The clipping parameter, ε (typically 0.2), which caps the ratio between 1-ε and 1+ε. This prevents massive, unstable weight updates.

Interpretation

Before looking at the reward, the Critic guesses: "Based on this prompt, I expect the model to get a score of around +1.0." If the actual combined reward turns out to be +2.5, the Advantage is +1.5 (2.5 - 1.0). This tells the model: "This response wasn't just good; it was better than expected."

Before we go thorugh the mathematics to understand the PPO loss function, lets calculate their derivatives.

PPO Derivative:

we only look at the unclipped portion of the objective function (which is active when the model update stays within safe boundaries).

The PPO objective function ($J_{\text{CLIP}}$) that we want to maximize is:

$$J_{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \left[ \min\left( r_t(\theta)\hat{A}_t, \, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t \right) \right]$$

Optimizers like Adam perform minimization (gradient descent), we define our PPO loss function ($L_{\text{PPO}}$) as the negative of the objective:

$$L_{\text{PPO}}(\theta) = - J_{\text{CLIP}}(\theta)$$

When the probability ratio $r_t(\theta)$ is unclipped, the min function evaluates simply to the unclipped term. Therefore, for an unclipped step, we differentiate: $$ \begin{aligned} L_{\text{PPO}}(\theta) &= -\hat{\mathbb{E}}_t \left[ r_t(\theta)\hat{A}_t \right] \\ &= -\hat{\mathbb{E}}_t \left[ \frac{\pi_{\theta}}{\pi_{\text{SFT}}} \hat{A}_t \right] \\ &= -\hat{\mathbb{E}}_t \left[ \frac{\hat{A}_t}{\pi_{\text{SFT}}} \pi_{\theta} \right] \\ \text{Taking derivative w.r.t. } \theta: \\ \nabla_\theta L_{\text{PPO}} &= -\hat{\mathbb{E}}_t \left[ \frac{\hat{A}_t}{\pi_{\text{SFT}}} \cdot \nabla_\theta \pi_\theta \right] \text{since $\hat{A}_t$ and $\pi_{\text{SFT}}$ are constants w.r.t. $\theta$} \\ &= -\hat{\mathbb{E}}_t \left[ \frac{\hat{A}_t}{\pi_{\text{SFT}}} \cdot \pi_\theta \cdot \nabla_\theta \log \pi_\theta \right] \text{(using the log-derivative trick)} \\ &= -\hat{\mathbb{E}}_t \left[ r_t(\theta) \cdot \hat{A}_t \cdot \nabla_\theta \log \pi_\theta \right] \text{(substituting $r_t(\theta) = \frac{\pi_\theta}{\pi_{\text{SFT}}}$)} \\ \text{Final derivative w.r.t. } \theta: \\ \nabla_\theta L_{\text{PPO}} &= - r_t(\theta) \cdot \hat{A}_t \cdot \nabla_\theta \log \pi_\theta \end{aligned} $$

Putting It All Together with maths

From above, the penalized reward, R = 2.4942.

Case 1 (Ratio is Clipped): When the derivative is zero

Let consider a simple example, where we have the following values:

Probability ratio rt(θ) = 1.2,
Advantage At = 1.5 (meaning this response performed better than expected), and
Clipping Parameter (ε): ε = 0.2 (meaning the ratio is capped between 1 - 0.2 = 0.8 and 1 + 0.2 = 1.2).

Now, we can calculate the PPO Loss:

$$ \begin{aligned} L_{PPO}(\theta) &= - \mathbb{E}_t \left[ \min\left(r_t(\theta)\hat{A}_t,; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t\right) \right] \\ &= - \min(1.2 \times 1.5,; \operatorname{clip}(1.2,0.8,1.2)\times 1.5) \\ &= - \min(1.8,; 1.2 \times 1.5) \\ &= - \min(1.8,; 1.8) \\ L_{PPO}(\theta) &= -1.8 \end{aligned} $$

Now, the derivative of the PPO Loss:

Since, rt(θ) = 1.2 is at the upper clipping boundary (1 + ε = 1.2), the derivative of the loss function with respect to θ is zero.

$$ \begin{aligned} \frac{\partial L_{\text{PPO}}(\theta)}{\partial \theta} &= 0 \end{aligned} $$

This means that the optimizer will not update the model parameters for this particular step, as it has reached the maximum allowed change in probability ratio.

Interpretation: This is a safety break, if a policy update tries to push the model too far too fast, PPO will stop updating the weight in this direction.

Case 2 (Ratio is Not Clipped): When the derivative is non-zero

Let consider a simple example, where we have the following values:

Probability ratio rt(θ) = 1.05, (It falls strictly between 0.8 and 1.2, so no clipping occurs.)
Advantage At = 1.5 (meaning this response performed better than expected), and
Clipping Parameter (ε): ε = 0.2 (meaning the ratio is capped between 1 - 0.2 = 0.8 and 1 + 0.2 = 1.2).

Now, we can calculate the PPO Loss:

$$ \begin{aligned} L_{PPO}(\theta) &= - \mathbb{E}_t \left[ \min\left(r_t(\theta)\hat{A}_t,; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t\right) \right] \\ &= - \min(1.05 \times 1.5,; \operatorname{clip}(1.05,0.8,1.2)\times 1.5) \\ &= - \min(1.575,; 1.2 \times 1.5) \\ &= - \min(1.575,; 1.8) \\ L_{PPO}(\theta) &= -1.575 \end{aligned} $$

Now, the derivative of the PPO Loss:

Since, rt(θ) = 1.05 is not at the clipping boundary (unclipped), the objective function depends directly on the model parameter, θ.

$$ \begin{aligned} L_{PPO}(\theta) &= -1.575 \times \frac{\partial \pi_{\theta} }{\partial \theta} \end{aligned} $$

The model weights (θ) are successfully updated by the gradient descent, so that next time it sees a similar prompt, it is sightly more likely to gnerate this preferred style of respone.