Katkat
Back

Apply reinforcement learning to product decisions

MetaData Scientist · Onsite
MediumData ScientistOnsiteMachine Learning

Last updated

This question assesses the ability to determine the appropriate RL framework for recommendation systems, evaluate policies using offline data, and implement safety guardrails.

Scenario

You are a Data Scientist at a large social media company. The product team wants to transition from a static supervised learning model—which predicts the probability of a click (CTR)—to an RL-based system for session-level feed ranking. The goal is to maximize long-term user retention rather than just immediate clicks.

Task

  1. Framework Selection: Compare and contrast using a Multi-Armed Bandit (MAB) approach versus a Markov Decision Process (MDP). Given that user preferences may evolve during a single session, which framework is more appropriate and why?

  2. Policy Evaluation: You plan to train the policy offline using historical logs. Explain how you would use Doubly Robust (DR) estimation to evaluate this new policy before deploying it. How do you check for sufficient overlap between the logging policy and your target policy?

  3. Reward Design & Guardrails: Retention is a delayed reward that is often sparse at the session level. How would you design a reward function that balances immediate engagement with long-term retention? Furthermore, propose two specific "guardrails" (safety checks) you would implement during an A/B test to ensure the new policy does not negatively impact user experience.

Hints

Hint 1

Consider whether the environment is stateless (MAB) or if the current action influences the future state of the user (MDP).

Hint 2

Doubly robust estimation combines a direct method (model-based) with importance sampling to reduce variance and bias.

Hint 3

Guardrails should focus on metrics that detect negative trends, such as increased report rates, decreased session duration, or high-variance changes in churn.

Follow-up questions

  • How would you handle the 'exploration-exploitation' trade-off during the initial online deployment phase?
  • If the reward signal is delayed by days, how would you construct a proxy reward that provides faster feedback during training?
  • What specific metrics would you monitor to detect 'policy drift' after the model has been deployed for a month?