Design an experiment to evaluate a new ads algorithm

You are a Product Analytics/Data Science partner for an ads ranking/recommendation team. Facebook has shipped (or plans to ship) a new ad recommendation algorithm and believes it is better.

Part A: How would you evaluate whether it is better?

Design an evaluation plan assuming you can run an experiment. In your answer:

• Define the goal and key stakeholders (advertisers, users, FB revenue).
• Propose a set of metrics with tradeoffs:
• Primary metric (choose one and justify)
• Diagnostic metrics to explain movement
• Guardrail metrics to prevent harm (user experience, platform integrity, advertiser outcomes)
• Choose the unit of randomization (e.g., user, session, advertiser) and explain interference risks (marketplace effects, repeated exposure, learning effects).
• Discuss basics of experiment execution: ramp plan, duration, power/MDE considerations, data quality checks (SRM), and how you would interpret heterogeneous impacts.

Part B: What if 50/50 randomization is not feasible?

Sometimes a 50/50 split is not appropriate (e.g., risk, capacity limits, model learning/feedback loops, or advertiser delivery constraints). Propose at least two valid alternatives for randomization or rollout (e.g., unequal allocation, phased ramp, cluster randomization, switchback, geo split), and explain:

• When each approach is appropriate
• Key pitfalls (bias, interference, novelty effects)
• How analysis changes (e.g., variance, sequential monitoring, CUPED)

Part C: If you cannot run a controlled test

If you are not allowed to run an online A/B test, how would you decide what to recommend to users and/or whether the new algorithm is better?

• Propose an approach using observational/offline data.
• Address confounding and selection bias.
• For recommendations (especially cold-start users), describe a reasonable strategy that balances relevance and exploration.

Be explicit about assumptions and failure modes.