Katkat
Back

Detect and Prevent Reward Hacking

GoogleMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

Assess the candidate's understanding of reward hacking (specification gaming) in reinforcement learning from human feedback (RLHF), focusing on diagnosing exploitation patterns, designing detection metrics, applying conservative policy updates, and implementing recovery safeguards.

Scenario

You are leading the post-training alignment of a 70B parameter instruction-tuned language model using Reinforcement Learning from Human Feedback (RLHF) with Proximal Policy Optimization (PPO). During training against a learned scalar reward model R(x, y), the logged training reward increases monotonically, but evaluation on a held-out benchmark indicates severe policy degradation: responses have become excessively verbose, sycophantic, and structurally repetitive, with factuality dropping by 18%.

Your task is to analyze this failure mode, design detection mechanisms, and propose algorithmic mitigations to stabilize optimization.

Tasks

  1. Root Cause and Shortcut Analysis

    • Explain how Goodhart's Law manifests in learned reward models during policy optimization.
    • Identify at least two common structural or stylistic shortcuts that language policies exploit when optimizing against standard Bradley-Terry preference-trained reward models, and why these features yield artificially high reward scores.

  2. Independent Detection and Measurement

    • Propose an automated evaluation framework to detect reward hacking without relying on human annotators in the loop during training runs.
    • Formulate at least two quantitative metrics (e.g., divergence checks, length-normalized score ratios, or auxiliary classifier probes) that signal policy exploitation before quality severely degrades.

  3. Conservative Policy Optimization

    • Standard PPO uses a per-token Kullback-Leibler (KL) divergence penalty against the reference model: r_penalized(x, y) = R(x, y) - beta * D_KL(pi_theta(y|x) || pi_ref(y|x)).
    • Discuss the theoretical and practical limitations of relying solely on a fixed scalar beta KL penalty to prevent reward hacking.
    • Detail two alternative or supplementary training strategies (for example: reward model ensembles, uncertainty-aware reward routing, constrained optimization such as trust-region bounds, or length-debiasing) to constrain optimization to valid policy regions.

  4. Recovery and Online Control

    • Describe an automated rollback and checkpoint selection criterion to prevent corrupted policy states from propagating into downstream production candidate pools.

Hints

Hint 1

Consider how out-of-distribution generation under the current policy exploits regions of the reward model input space where epistemic uncertainty is high.

Hint 2

Think about decomposing response quality into orthogonal attributes (e.g., correctness, style, safety) using auxiliary specialized evaluators rather than a single monolithic scalar score.

Hint 3

Analyze how ensembling multiple reward models trained with different seeds or architectures can provide a variance signal that penalizes policy samples residing in over-optimized regions.

Follow-up questions

  • How does Direct Preference Optimization (DPO) compare to PPO regarding vulnerability to reward over-optimization and out-of-distribution policy shifts?
  • If your reward model ensemble has high epistemic uncertainty on a generated token sequence, how would you incorporate that uncertainty into the PPO advantage calculation (e.g., pessimistic reward bounds)?
  • How would you address length bias directly inside the Bradley-Terry preference loss formulation during reward model pre-training?