Katkat
Back

Design an interference-robust A/B test for monetization

TikTokData Scientist · HR Screen
HardData ScientistHR ScreenAnalytics & Experimentation

Last updated

This question evaluates experimental design and causal inference skills within Analytics & Experimentation, emphasizing interference mitigation, clustering and randomization choices, eligibility and exposure rules, metric hierarchy and guardrails, power and duration planning, variance-reduction techniques, and cluster-robust inference in a two-sided marketplace. It is commonly asked to test a data scientist's practical-application ability to balance monetization and growth trade-offs through precise statistical planning, pre-registered stopping rules, SRM detection, MDE and rollout decision thresholds rather than purely conceptual understanding.

You’re launching a new tipping UI on creator (PGC/OGC) posts to motivate monetization without hurting growth or traffic. Design an A/B test that is robust to interference and supply–demand dynamics. Specify: 1) randomization unit and clustering strategy (e.g., creator-level, ego-network, geo-level) to mitigate cross-user and cross-post spillovers; 2) eligibility and exposure rules to prevent treatment contamination across US and Asia time zones; 3) primary metric hierarchy (e.g., payer conversion per DAU, ARPPU, creator revenue share) and guardrails (retention, session length, abuse reports, ad revenue cannibalization); 4) power and duration targeting at least one weekly cycle, ramp plan, and SRM detection with pre-registered stop rules; 5) variance reduction (CUPED covariates such as pre-experiment spend and creator popularity) and cluster-robust inference; 6) decision thresholds, and how you’d roll out if treatment helps monetization but slightly hurts growth. Be specific about exact formulas and the minimal detectable effect you target.