Katkat
Back

Investigate Traffic Distribution Impact on Retention Decrease

TikTokData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

Evaluates A/B test diagnostics for a retention decrease after a button-color change. Strong answers check allocation, balance, logging, maturation, statistical robustness, segments, and decision criteria before blaming treatment.

Scenario

An A/B test changed a button’s color from green to red; retention dropped. Stakeholders suspect traffic allocation problems.

Question

How would you investigate whether the traffic distribution between control and treatment caused the retention decrease? Which diagnostics, balance checks, or statistical tests would you run before concluding the new color hurts retention?

Hints

Discuss randomization sanity checks, covariate balance, sequential testing bias, and segment-level retention comparisons.