Experiment Design for Exploratory Recommendations

You are launching an online A/B test for a new recommendation algorithm. The goal is to increase users' exploration behavior, meaning discovery of new or diverse content. A known challenge is that the new algorithm may slightly reduce core short-term engagement metrics such as like rate.

Design the experiment, define exploration and long-term satisfaction metrics, and make a launch recommendation when the experiment window is too short to fully observe long-term impact.

Constraints & Assumptions

• Randomization can be user-level unless network effects require clustering.
• Exploration should be valuable discovery, not random novelty.
• Short-term engagement losses may be acceptable only if long-term value is credible.
• Include retention, satisfaction, creator ecosystem, and safety guardrails.

Clarifying Questions to Ask

• What content universe and surfaces does the recommender affect?
• What is the acceptable short-term engagement trade-off?
• What long-term outcomes are historically linked to exploration?
• Are survey, retention, and creator-side outcomes available?

Part 1 - Experiment Design

How would you design and run the experiment to decide whether to launch?

What This Part Should Cover

• Define treatment, control, eligibility, randomization unit, exposure, sample size, duration, and analysis plan.
• Use user-level assignment with sticky variants; consider clusters if social spillovers are material.
• Predefine decision rules balancing exploration lift and engagement guardrails.
• Include SRM, logging checks, novelty, and segment analysis.

Part 2 - Metric Definition

How would you define exploration, long-term satisfaction, and ecosystem health?

What This Part Should Cover

• Exploration metrics may include content/category diversity, novelty, new creator discovery, entropy, distance from historical interests, and repeat engagement with discovered content.
• Satisfaction metrics may include surveys, long-clicks, retention, return rate, session quality, negative feedback, and durable engagement.
• Ecosystem metrics may include creator distribution, new creator exposure, safety, revenue, and content quality.

Part 3 - Limited Duration Decision

If the experiment window is limited and long-term impact is not fully observed, how would you analyze and recommend?

What This Part Should Cover

• Use validated leading indicators, proxy-to-long-term relationships, cohort early signals, and historical backtests.
• Quantify uncertainty and downside risk.
• Recommend ship, do not ship, ramp gradually, or extend test based on thresholds.
• Use long-term holdouts or phased rollout to continue learning.