Explain Treatment Results and Recommend Launch Criteria for Experiments
UpstartData Scientist · Technical Screen
Last updated
Explain Treatment Results and Recommend Launch Criteria for Experiments evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Scenario
Evaluating two treatments (t1, t 2) on Gross Booking (GB) and Variable Consideration (VC)
Question
Treatment t1 shows no significant change in GB or VC, while t2 shows a significant increase in GB but a significant decrease in VC. How would you explain these results to the PM and recommend next steps? Given confidence intervals GB [+0.1%, +2.3%] with +$0.48 lift, and VC [-2.5%, -1.5%] with -$0.20 loss, would you launch t2? Justify your decision. How would you segment users or orders to identify cohorts with positive GB impact and no negative VC impact? If 20 different experiments run simultaneously, how would you define launch criteria to control false discoveries and ensure reliable decisions?
Hints
Discuss trade-offs, statistical power, cost-benefit, cohort analysis, multiple-testing corrections (e.g., FDR).
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.