How to analyze Simpson's paradox
LinkedInData Scientist · Technical Screen
Last updated
This question evaluates competency in causal reasoning, statistical inference, experimental design, and detection of Simpson's paradox within the Analytics & Experimentation domain.
A marketing team wants to evaluate a new email campaign. Two email versions, A and B, were tested over two weeks in two cities: San Francisco and New York. The overall pooled data suggests version A has a higher conversion rate than version B, but when you break the results down by city, and possibly by week, version B appears better in every subgroup.
How would you determine whether the new email is actually better? In your answer, discuss:
- what metric(s) you would define as primary and which guardrail metrics you would monitor,
- how Simpson's paradox can arise in this setting,
- what confounders or sources of imbalance you would check,
- how you would re-analyze the existing data,
- whether and how you can compute confidence intervals, and
- how you would redesign the experiment if the original allocation across cities or time was imbalanced.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.