Katkat
Back

Estimate impact without experiments and pick variant

UpstartData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question evaluates causal inference and experimental-analysis competencies in Analytics & Experimentation and Data Science, covering observational estimands, causal identification assumptions and biases, uncertainty quantification for A/B/C tests, multiple-comparisons reasoning, and post-launch forecasting and monitoring.

Part A — Measuring impact when you cannot run an experiment

You are a Staff Data Scientist working on a product change (feature/policy/model update). Stakeholders want to measure causal impact (incremental lift) of the change, but you cannot launch a randomized experiment (e.g., legal constraints, all users must receive the change, platform limitations, or risk).

Task:

  1. Propose an end-to-end approach to estimate the causal impact of the change using observational data (you may use an ML-based counterfactual if appropriate).
  2. Clearly state:
  • The estimand (e.g., ATE, ATT, incremental purchases per user/day).
  • Key assumptions required for causal identification.
  • Major biases/failure modes (confounding, selection bias, interference, data drift, novelty effects, etc.).
  • How you would validate the approach (placebo tests, negative controls, sensitivity analysis, backtesting).
  1. Explain how you would reason about short-term vs. long-term impact, and what additional data or modeling you would need.

Part B — 3-variant experiment and forecasting post-launch conversion

You ran an online experiment with three variants (A/B/C). The goal is to maximize CTP (purchase rate), defined as:

[ \mathrm{CTP} = \frac{#\text{purchases}}{#\text{visits}} ]

Observed results (assume visits are independent Bernoulli trials; one purchase at most per visit):

  • Variant A: 150 visits, 43 purchases
  • Variant B: 200 visits, 48 purchases
  • Variant C: 100 visits, 15 purchases

Questions:

  1. Which variant is “winning”?
  • Provide point estimates of CTP.
  • Quantify uncertainty (e.g., confidence/credible intervals).
  • Address multiple comparisons / decision criteria if needed.
  1. Suppose you launch the chosen variant to 100% traffic. How would you predict the future CTP after launch?
  • Describe a statistical approach to generate a forecast (and interval).
  • List key factors that may cause post-launch CTP to differ from experiment CTP (traffic mix shift, seasonality, ramp-up effects, novelty, instrumentation changes, etc.).
  • Mention how you would monitor/validate the forecast after launch (guardrails, alerting, recalibration).