Katkat

Evaluate impact without randomized experiments

UberData Scientist · Technical Screen

Last updated

This question evaluates a data scientist's competence in causal inference and identification strategies for estimating promotion effects from observational longitudinal data, emphasizing challenges such as staggered rollout, interference (spillovers), seasonality, and uncertainty quantification.

If you cannot run an experiment, propose and justify at least three identification strategies to estimate the promotion’s causal effect, including assumptions, diagnostics, and failure modes: (1) Difference‑in‑Differences/event‑study with staggered rollout using modern estimators (avoid naive TWFE bias), parallel‑trends checks, and placebo tests; (2) Synthetic Control at the geo level with pre‑period fit metrics and sensitivity to donor pool/regularization; (3) Instrumental Variables exploiting exogenous variation (e.g., eligibility rules, random delivery outages), with relevance/exclusion checks and weak‑IV diagnostics; (4) Regression Discontinuity if eligibility thresholds exist (e.g., tenure ≥ 30 days), with bandwidth/functional‑form sensitivity and McCrary density tests; (5) Propensity‑score or doubly‑robust learners (IPW/AIPW, causal forests) with overlap checks and covariate balance. Explain how you would report ATE/ATT with uncertainty, account for interference/network effects, and control for seasonality and concurrent campaigns.