Katkat
Back

Quantify impact without an A/B test

StripeData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design a quasi-experimental study to estimate causal impact when randomized controlled trials (RCTs) are unavailable, specifically focusing on observational causal inference techniques.

Scenario

A payment-success optimization was rolled out to 100% of merchants in the 'EMEA' region last Monday to reduce failed transactions. Management wants to know the revenue lift attributable to this change. However, no holdout group was created, and you have limited time.

Your task is to design an analysis to estimate the causal impact of this feature on Total Payment Volume (TPV) using historical data.

Task

  1. Propose an identification strategy (e.g., Difference-in-Differences, Synthetic Control, or Interrupted Time Series) to estimate the lift.
  2. Explain how you would select a 'control' group (if applicable) and how you would account for confounding factors like regional seasonality or the current holiday shopping period.
  3. Define a decision rule: Based on the confidence interval of your estimate, what result would warrant a 'Ship,' 'Roll Back,' or 'Iterate/Deep Dive' decision?

Constraints & Considerations

  • You have access to daily TPV data for all regions.
  • Assume merchant behavior in EMEA might be correlated with behavior in other regions (e.g., NA, APAC).
  • Address how you would validate that your chosen control group is valid (e.g., pre-treatment trend alignment).
  • Keep the complexity manageable for a 6-hour turnaround time.

Hints

Hint 1

Consider using Difference-in-Differences (DiD) by pairing EMEA with a region that exhibits similar historical trends but did not receive the update.

Hint 2

Use a 'placebo test' or a pre-treatment period analysis to check if the treatment and control regions move in parallel before the launch date.

Hint 3

If the treatment region is unique, consider a Synthetic Control Method approach where you construct a weighted average of other regions to mimic EMEA’s pre-launch revenue patterns.

Follow-up questions

  • How would you adjust your analysis if a large marketing campaign launched in the EMEA region simultaneously with your feature update?
  • What are the risks of 'Selection Bias' if only high-volume merchants in EMEA were affected by the infrastructure change?
  • If you found a statistically significant lift, how would you distinguish between an increase in 'payment success' versus an increase in 'merchant transaction volume'?