Direct Policy Optimization (DPO): Reinforcement Learning with Human Feedback (with the maths behind fine-tuning LLMs)

1. Overview & Context

When we fine-tune a Large Language Model using standard next-token prediction, we teach it grammar, syntax, and basic pattern matching. But raw text generation alone doesn't guarantee that an LLM will be helpful. To truly align a model with human intent, we need a mechanism to reward desirable behavior and discourage toxic or unhelpful outputs.

With Reinforcement Learning with Human Feedback (RLHF) and its optimization algorithm, Direct Policy Optimization (DPO), we can help align the model with human preferences with the help of Reward Modeling.

In the previous blog, Proximal Policy Optimization: Reinforcement Learning with Human Feedback, we explored the complete mathematical and architectural pipeline behind RLHF with PPO.

In this blog, we will explore the complete mathematical and architectural pipeline behind RLHF with DPO, and how it differs from PPO.

2. Why DPO?

As we have already seen in previous blog, we need 4 models for alignemtn in PPO, namely, Reference Model (SFT LLM), Actor (Policy LLM), Reward Model (adapted from SFT), and , Critic Model(Value Function LLM). Training all of these models can be computationally expensive and complex.

DPO simplify this process by directly optimizing the policy model directly on human preferences without the need for a separate reward model or critic LLM.

Direct Policy Optimization (DPO) offers several advantages over traditional Reinforcement Learning with Human Feedback (RLHF) approaches like Proximal Policy Optimization (PPO):

• Simpler Implementation: DPO eliminates the need for complex reward modeling and policy gradient estimation.

• Reduced Variance: By directly optimizing the policy, DPO can reduce the variance in the gradient estimates.

• Better Sample Efficiency: DPO can achieve better performance with fewer samples compared to PPO.

3. Direct Policy Optimization (DPO)

We can rewrite the Reward Model's scoring function directly in terms of the LLM's own token probabilities. Therefore, instead of using a separate reward model to evaluate the response, DPO looks at the LLM itself, and evaluates, how likely is the policy llm to generate a choosen response compared to our reference LLM model, and similarly, how likely is the policy llm to generate a rejected response compared to the reference LLM model.

Therefore, we only need two models: the policy model and the reference model.

                        ┌─────┐
            Prompt ───► │ LLM │───► Response ───► DPO Loss Function ───► Parameter Update
                        └─────┘
            

The DPO Loss Function mathematically forces the model to do two things at once:

• Maximize the probability of the Chosen response relative to the Reference model.

• Minimize the probability of the Rejected response relative to the Reference model.

The standard naming conventions are the same as in the previous blog on PPO:

3. Human Preference Data

To train the reward model, we use pairwise human preference data, like we did for reward model training in PPO:

• Collect Prompts :The instruction or question given to the model.

• Chosen Response (y_w) : The preferred, high-quality answer (written by a human or a better model).

• Rejected Response (y_l) : The dispreferred, lower-quality answer (unhelpful response generated by the SFT model).

Example Preference Data Structure

{
"prompt": "x: What is 2 + 2?",
"chosen": "y_w: The answer to 2 + 2 is 4.",
"rejected": "y_l: 2 + 2 is 5, but sometimes math is relative."
}

4. Fine-tuning the Policy with DPO

In this section, we will explore how the Policy LLM is fine-tuned using Direct Policy Optimization (DPO) to align it with human preferences.

Here, we combine everything into DPO training step to update LLM weights (θ).

        ┌──────────┐
y_w ───►│Policy LLM│ ───► log probabilities
    \  /│  (π_θ)   │
     \/ └──────────┘
     /\  
    /  \┌──────────┐
y_l ───►│ SFT LLM  │ ───► log probabilities
        │ (π_sft)  │
        └──────────┘
                

Policy LLM Architecture

The Policy LLM isn't built from scratch. It is initialized directly from our trained SFT checkpoint (π_SFT) without any modifications.

Calculate the Implicit Rewards

It checks how confident the policy model π_θ is, about this response compared to the reference model π_SFT.

DPO Loss Function

The DPO algorithm uses a simple standard sigmoid cross-entropy loss function to update the policy parameters:

Derivation of the DPO Loss Function

The DPO loss function is derived from the principle of maximizing the likelihood of the chosen response while minimizing the likelihood of the rejected response. The sigmoid function is used to convert the difference in scores into a probability, which is then used in the cross-entropy loss.

The Weighting Factor: If the model is already doing a good job ranking chosen over rejected, the sigmoid term approaches 1, making the weighting factor approach 0 (stopping unnecessary updates). If the model gets it wrong, the gradient spikes, forcing a heavy correction.

Mathematical and Numerical Example

Lets say, we pass our prompt and responses through both models, π_SFT and π_θ, yielding the following sequence log-probabilities log π:

If the model gets it right: As σ approaches 1.0 (meaning the model strongly prefers the chosen response), the term (1 - σ) approaches 0, No unnecessary gradient updates.

If the model gets it wrong: The gradient scales up, forcefully tweaking the internal weights (θ) to increase the probability of generating y_w while decreasing the probability of generating y_l.