Katkat
Back

Compare Language-Model Post-Training Methods

GoogleMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question tests your conceptual depth in LLM alignment by asking you to differentiate between supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), and group-relative optimization (e.g., GRPO) based on their mathematical objectives and operational characteristics.

The Problem

Modern large language models typically undergo a post-training phase to align the base model with human intent. While these methods share the same goal, they differ significantly in their optimization objectives, computational requirements, and sensitivity to data quality.

Compare the following four alignment techniques:

  1. Supervised Fine-Tuning (SFT)
  2. RLHF (PPO-based)
  3. Direct Preference Optimization (DPO)
  4. Group-Relative Policy Optimization (GRPO)

Provide a structured comparison covering:

  • The Objective Function: Explain the core loss function for each. How do they treat the reward signal differently (e.g., implicit vs. explicit)?
  • Data Requirements: What type of data does each require (e.g., demonstration pairs, preference pairs, or scalar reward models)?
  • Stability and Complexity: What are the primary failure modes for each? Specifically, discuss the stability challenges inherent in the RL-based methods versus the offline nature of preference-based methods.
  • Computational Efficiency: Compare the memory and infrastructure requirements for training, especially regarding the need for reference models or auxiliary reward models.

Hints

Hint 1

Consider whether the method requires a separate reward model or if the preference signal is incorporated directly into the language model update.

Hint 2

Think about the difference between KL-regularized optimization objectives and simple cross-entropy minimization.

Hint 3

Contrast the 'online' nature of sampling during training (PPO/GRPO) with 'offline' training where the model updates based on a static dataset of preference pairs.

Follow-up questions

  • How does the issue of 'reward hacking' manifest differently in RLHF versus DPO?
  • Under what data constraints would you choose SFT over DPO, and why?
  • If you were deploying a model that requires high reasoning capabilities, why might GRPO be preferred over standard DPO?