Katkat

Estimate impact of global launch without holdout

AirbnbData Scientist · Technical Screen

Last updated

This question evaluates a candidate's competence in causal inference, observational study design, and statistical uncertainty quantification for product analytics, specifically estimating lift after a global launch without a holdout while addressing confounding, interference, and operational data-quality issues.

A product feature was launched globally on 2025-05-10 with no control or holdout. Design a plan to estimate its causal lift on weekly revenue and 7-day retention. Propose at least two independent identification strategies (e.g., interrupted time series with covariates, synthetic control across markets, difference-in-differences using unaffected cohorts, regression discontinuity at the launch timestamp, IV/front-door using eligibility or latency shocks). For each, specify: (a) identification assumptions and threats; (b) unit of analysis, data needs, and pre-period length; (c) diagnostics and falsification tests (placebo dates, negative/positive control outcomes, pre-trend checks); (d) uncertainty quantification and 95% CIs (delta method vs bootstrap, cluster level); (e) how you will bound effects if assumptions partially fail (e.g., amplification/robustness curves, sensitivity to unobserved confounding); (f) how you will handle interference/spillovers, seasonality/holidays, concurrent marketing, logging/schema changes, backfilled events, selection into exposure, and migration; (g) a decision framework to reconcile conflicting estimates and produce a single recommendation (ship, rollback, or iterate), including acceptable risk thresholds.