Design robust A/B test with interference and seasonality
TikTokData Scientist · Technical ScreenLast updated
This question evaluates expertise in experimental design, causal inference, statistical power and minimum detectable effect calculation, variance-reduction techniques, sequential monitoring, and diagnostics for interference, spillovers, and weekly seasonality in A/B testing.
You are launching a redesigned onboarding flow expected to increase Day-7 activation but may cause network effects (users invite others) and weekly seasonality. Design an experiment plan that covers: (1) hypothesis, primary metric(s), guardrail metrics, and exact metric definitions with attribution windows; (2) unit of randomization and exposure (user, household, geo, or cluster) and why, given potential interference; (3) sample size and power analysis, target MDE, duration assumptions, and how you’ll account for seasonality (e.g., full-week multiples); (4) variance reduction (e.g., CUPED with pre-period covariates), stratification, or geo-matched pairs; (5) SRM detection and remediation plan; (6) sequential monitoring approach and stopping rules (alpha spending) to avoid p-hacking; (7) a ramp plan with holdouts and a plan for novelty/learning effects; (8) diagnostics for noncompliance, bot traffic, and triggered vs. assigned populations; (9) how you’d detect and mitigate interference/spillovers (cluster randomization, geo experiments, or switchback) and quantify any bias if user-level randomization is used; (10) interpretation plan if primary and guardrail metrics disagree, and how you’d decide to ship.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.