Explain SFT, DPO, PPO, and the Role of KL Regularization

Explain the objectives and practical differences among supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and PPO-based language-model post-training.

Part 1 — SFT and the Role of Demonstrations

Explain the SFT training objective and why a workflow may still use SFT before preference optimization.

What This Part Should Cover

Conditional token likelihood on demonstrations, loss masking, and the distinction between learning demonstrated behavior and comparing preferred alternatives.

Part 2 — DPO and PPO

Explain what standard DPO optimizes and how its training workflow differs from PPO-based reinforcement learning with preference-derived rewards. Discuss why SFT and DPO may remain useful rather than assuming one method always replaces the others.

What This Part Should Cover

Chosen/rejected response pairs, the reference policy, relative log probabilities, and the roles of sampling, reward estimation, and policy updates.

Part 3 — The KL Question

Assess the statement that DPO does not need KL regularization. Distinguish the absence of a separately computed KL penalty in the usual DPO loss from the KL-regularized derivation and reference-policy dependence.

What This Part Should Cover

A precise correction of the premise and the role of the DPO temperature or regularization parameter.

Constraints

Discuss standard formulations rather than claiming that every later variant behaves identically. Use an autoregressive language model, demonstration data for SFT, and response-preference pairs for DPO. No task-specific winner is assumed.

Clarifying Questions

• Are good demonstrations, preference pairs, or an online reward signal available?
• Is the question about the training loss, the full data-collection workflow, or deployment quality?