Katkat
Back

Resolve Conflicting A/B Test Results in Cities

LinkedInData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

Evaluates interpretation of conflicting A/B test results across cities using Simpson's paradox and stratified estimands. Strong answers check weighting, SRM, heterogeneity, confidence intervals, and rollout criteria.

Scenario

An A/B test was run in City X and City Y. In each city variant A beats B, yet when combined across both cities variant B looks better.

Question

How would you interpret these conflicting results?

What decision would you make and what additional analysis is needed before rollout?

Hints

Discuss Simpson’s paradox, weighting by traffic, heterogeneous treatment effects, stratified analysis, and checking sample size.