Katkat
Back

Resolve Simpson’s paradox in A/B email test

LinkedInData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to identify and resolve Simpson's Paradox in A/B test results, where an aggregate trend disappears or reverses when segmented by user cohorts.

Scenario

LinkedIn is testing a new "Premium Upsell" email design to increase the click-through rate (CTR). You are given the following aggregate performance data:

GroupSentClickedCTR
Control10,0005005.0%
Variant10,0004504.5%

At first glance, the Control group performs better. However, a junior analyst suggests looking at the data segmented by user seniority level (Junior vs. Senior):

Junior Users:

GroupSentClickedCTR
Control8,0003204.0%
Variant2,000904.5%

Senior Users:

GroupSentClickedCTR
Control2,0001809.0%
Variant8,0003604.5%

Task

  1. Explain why the aggregate CTR for the Variant (4.5%) appears lower than the Control (5.0%) despite the Variant outperforming the Control in both individual segments.
  2. Identify the confounding variable here. How did the imbalance in user distribution affect the aggregate results?
  3. Propose a method to calculate the "true" impact of the email design that accounts for the seniority distribution differences.

Hints

Hint 1

Look closely at the distribution of Senior users between the two groups; is the randomization balanced?

Hint 2

Simpson's Paradox occurs when a confounding variable influences both the group assignment and the outcome metric.

Hint 3

Consider using weighted averages or post-stratification techniques to normalize the results based on the target population distribution.

Follow-up questions

  • If you discovered this imbalance after the experiment finished, would you trust the segment-level results or the aggregate results? Why?
  • How would you adjust your experimental design process (e.g., randomization strategy) to prevent this kind of imbalance in future tests?
  • If the 'Senior' segment had significantly higher engagement overall regardless of the email, how would you statistically control for this during the analysis phase?