Explain and validate A/B test assumptions
UberData Scientist · Technical ScreenLast updated
This question evaluates proficiency in experimental design, causal inference, and statistical diagnostics for A/B testing, covering competencies such as randomization integrity, SUTVA/interference, noncompliance, time-varying effects, metric missingness, sequential testing, and heterogeneous treatment effects.
List the core assumptions required for a valid online A/B test and, for each, describe: (1) what the assumption formally means; (2) one realistic product scenario where it is violated; (3) a diagnostic you would run to detect the violation; and (4) a concrete mitigation/redesign. Cover at least: randomization integrity and sample ratio mismatch (SRM), independence/SUTVA and interference (e.g., network effects, shared inventory), stable unit exposure and cross-over/noncompliance, stationarity/time trends and novelty/learning effects, metric logging bias/missingness (MCAR/MAR/MNAR), sequential peeking and error inflation, and heterogeneous treatment effects across key segments. For each assumption, specify exactly how you would implement the check (e.g., which statistical test or visualization, which pre-experiment covariates, how long a pre-period, whether to use cluster randomization, stratification, CUPED, or staggered ramps).
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.