Estimate impact of global launch without holdout
AirbnbData Scientist · Technical ScreenLast updated
This question evaluates a candidate's competence in causal inference, observational study design, and statistical uncertainty quantification for product analytics, specifically estimating lift after a global launch without a holdout while addressing confounding, interference, and operational data-quality issues.
A product feature was launched globally on 2025-05-10 with no control or holdout. Design a plan to estimate its causal lift on weekly revenue and 7-day retention. Propose at least two independent identification strategies (e.g., interrupted time series with covariates, synthetic control across markets, difference-in-differences using unaffected cohorts, regression discontinuity at the launch timestamp, IV/front-door using eligibility or latency shocks). For each, specify: (a) identification assumptions and threats; (b) unit of analysis, data needs, and pre-period length; (c) diagnostics and falsification tests (placebo dates, negative/positive control outcomes, pre-trend checks); (d) uncertainty quantification and 95% CIs (delta method vs bootstrap, cluster level); (e) how you will bound effects if assumptions partially fail (e.g., amplification/robustness curves, sensitivity to unobserved confounding); (f) how you will handle interference/spillovers, seasonality/holidays, concurrent marketing, logging/schema changes, backfilled events, selection into exposure, and migration; (g) a decision framework to reconcile conflicting estimates and produce a single recommendation (ship, rollback, or iterate), including acceptable risk thresholds.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.