Katkat

Measure rider incentive causal ROI

UberData Scientist · Technical Screen

Last updated

This question evaluates a data scientist's competency in causal inference and experimental design, marketplace health metric construction, ROI estimation, and the handling of selection bias, spillover effects, multiple testing, and heterogeneity in two-sided platforms.

You plan a rider‑side incentive (e.g., 20% off up to $10) targeted by a propensity model. Estimate causal incrementality and ROI despite selection and marketplace spillovers.

Do the following:

  • Define marketplace‑health metrics with formulas: matching quality (pickup ETA p50, post‑dispatch cancel rate within 5 minutes, driver idle minutes/trip), demand (incremental trips, GMV lift), supply (acceptance rate, earnings/hour), and a composite quality‑of‑match metric (distance‑to‑pickup p50).
  • Propose identification in three scenarios: (A) randomized geo holdouts; (B) thresholded offer scores enabling regression discontinuity (specify bandwidth selection, continuity checks, McCrary test, and polynomial order); (C) business‑rule targeting enabling IV via an operational friction instrument (state relevance and exclusion, and what threats you’ll test).
  • Write the ROI formula including cannibalization, subsidy burn, surge/ETA externalities, and habit formation. Specify how to estimate LTV lift and amortize CAC; include confidence intervals via the delta method or bootstrap.
  • Design a two‑stage randomization (city×week, then rider) to identify direct vs spillover effects; define estimands for total, direct, and indirect effects.
  • Multiple testing: choose and justify a correction strategy (Bonferroni vs BH vs hierarchical testing) for 1 primary, 3 key secondaries, and ~20 diagnostics.
  • Heterogeneity: outline a pre‑registered plan to detect effect moderation by city tier and weather using causal forests or group‑wise models while controlling Type‑S/M errors.
  • Diagnostics: enumerate pre‑trend checks, placebo windows, and negative‑control outcomes; define criteria for declaring success and for rollback.