Katkat
Back

Compare GRPO and PPO and Explain Sparse Versus Dense Rewards

SiemensMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question evaluates your understanding of modern reinforcement learning post-training techniques for language models, focusing on GRPO versus standard PPO and the trade-offs between sparse outcome rewards and dense process supervision.

Scenario

You are designing the reinforcement learning post-training pipeline for an LLM that solves multi-step engineering and physical reasoning problems (e.g., verifying finite element stress limits or code synthesis for automated industrial controllers).

You are evaluating two RL algorithms: Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), alongside two reward modeling strategies: Outcome-based Reward Models (ORMs) and Process-based Reward Models (PRMs).

Questions to Address

1. Advantage Estimation: Critic vs. Group Relative

  • Detail how standard PPO computes advantages using a separate learned value model (critic) and Generalized Advantage Estimation (GAE).
  • Detail how GRPO eliminates the value network by sampling a group of G outputs for a single prompt and standardizing rewards across the group.
  • What happens mathematically and empirically in GRPO if all G outputs receive identical rewards (e.g., all succeed or all fail)?

2. Computational and Memory Trade-offs

  • Compare the GPU memory footprints and training throughput between PPO (actor, reference model, critic, reward model) and GRPO.
  • Discuss the trade-off introduced by replacing critic inference with generating G independent completions per prompt.

3. Sparse (Outcome) vs. Dense (Process) Supervision

  • Define outcome supervision (sparse terminal reward) versus process supervision (dense step-level reward) in multi-step reasoning.
  • Explain the credit assignment problem associated with sparse rewards on long reasoning chains.
  • Compare the labeling cost, reward hacking risks, and verifiability of ORMs versus PRMs in deterministic engineering domains.

4. System Recommendation

Given an automated physical simulation environment that can execute and deterministically grade final code or outputs in under 200 ms, justify your choice between GRPO vs. PPO and sparse vs. dense rewards for scalable training.

Hints

Hint 1

Recall that GRPO computes advantages by normalizing rewards within a group: `A_i = (r_i - mean(r)) / (std(r) + eps)`.

Hint 2

Consider the GPU memory required by a critic model of comparable parameter size to the policy, especially under distributed FSDP/ZeRO setups.

Hint 3

Think about whether intermediate steps in deterministic engineering tasks can be verified programmatically or if they require a noisy learned reward model.

Follow-up questions

  • How does increasing the group size G in GRPO affect rollout latency versus gradient variance reduction?
  • If using a Process Reward Model (PRM), how do you prevent the policy from learning 'style hacks' or generating unnecessary padding steps to accumulate positive step rewards?
  • Can GRPO be adapted to process-level rewards, and if so, how would you structure the group baseline at the token or step level?