Measure rider incentive causal ROI
UberData Scientist · Technical ScreenLast updated
This question evaluates a data scientist's competency in causal inference and experimental design, marketplace health metric construction, ROI estimation, and the handling of selection bias, spillover effects, multiple testing, and heterogeneity in two-sided platforms.
You plan a rider‑side incentive (e.g., 20% off up to $10) targeted by a propensity model. Estimate causal incrementality and ROI despite selection and marketplace spillovers.
Do the following:
- Define marketplace‑health metrics with formulas: matching quality (pickup ETA p50, post‑dispatch cancel rate within 5 minutes, driver idle minutes/trip), demand (incremental trips, GMV lift), supply (acceptance rate, earnings/hour), and a composite quality‑of‑match metric (distance‑to‑pickup p50).
- Propose identification in three scenarios: (A) randomized geo holdouts; (B) thresholded offer scores enabling regression discontinuity (specify bandwidth selection, continuity checks, McCrary test, and polynomial order); (C) business‑rule targeting enabling IV via an operational friction instrument (state relevance and exclusion, and what threats you’ll test).
- Write the ROI formula including cannibalization, subsidy burn, surge/ETA externalities, and habit formation. Specify how to estimate LTV lift and amortize CAC; include confidence intervals via the delta method or bootstrap.
- Design a two‑stage randomization (city×week, then rider) to identify direct vs spillover effects; define estimands for total, direct, and indirect effects.
- Multiple testing: choose and justify a correction strategy (Bonferroni vs BH vs hierarchical testing) for 1 primary, 3 key secondaries, and ~20 diagnostics.
- Heterogeneity: outline a pre‑registered plan to detect effect moderation by city tier and weather using causal forests or group‑wise models while controlling Type‑S/M errors.
- Diagnostics: enumerate pre‑trend checks, placebo windows, and negative‑control outcomes; define criteria for declaring success and for rollback.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.