Design a feed ads A/B test with guardrails
MetaData Scientist · Onsite
Last updated
This question evaluates a data scientist's competency in experimental design, causal inference, statistical power analysis, and product-metric trade-offs within the Analytics & Experimentation domain, and is commonly asked to assess the ability to increase monetization while protecting user and creator experience.
You plan to insert one extra ad every 8 organic posts in the main feed. Design the experiment: (a) unit of randomization (user-level vs session-level) and why, (b) primary metrics (ad revenue per user) and guardrails (session length, 7-day retention, creator impressions, complaint rate), (c) power analysis given a +2.0% revenue MDE and 0.8% SD at user-day granularity, (d) SRM checks and how you’d detect novel-ad novelty effects, (e) ramp strategy and geographic/age holdouts, and (f) a long-term holdback to catch delayed churn. How will you address network spillovers (users sharing screenshots of ads) and avoid biased estimates from dynamic ranking reacting to the treatment?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.