Katkat
Back

Design A/B test and success metrics for new feature

MetaData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This question evaluates a data scientist's competency in experimental design, metric selection and guardrails, power and sample-size calculations, logging and attribution specifications, and revenue impact estimation for a new feature on a social media platform.

You are proposing a new Instagram feature: shareable Collections 2.0 meant to increase creator activity and monetization. Define success and guardrails:

  • State the mission and primary North Star metrics (one for engagement, one for revenue) and at least three counter metrics to protect experience and integrity. Design the experiment:
  • Choose user-level or geo-clustered A B test and justify considering network effects and contamination. Propose a stratification scheme using deciles of an engagement score and outline the randomization and ramp plan.
  • With baseline D30 retention 30% and a target 1% relative lift, compute the required sample size per arm at 90% power and alpha 0.05. State assumptions and formula.
  • Define stopping rules, pre-registration, and guardrail thresholds such as crash rate and time spent. Measurement and decisioning:
  • Specify the exact logging spec and how you would attribute revenue. Given average 4 ad impressions per DAU, CTR 1%, and revenue per ad click 0.50 dollars, estimate weekly incremental revenue if the feature increases ad clicks by 2% on 100 million DAUs. Include sensitivity and uncertainty bands.
  • List reasons not to launch even with statistically significant wins, including technical feasibility and long-term costs.