Katkat
Back

Diagnose Traffic Allocation in A/B Test Results

TikTokData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Diagnose Traffic Allocation in A/B Test Results states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

Online experimentation and causal analysis for a consumer app.

Question

An A/B test changed a call-to-action button from green (control) to red (treatment) and retention dropped. Describe the diagnostics you would run to decide whether uneven traffic allocation or other experiment-quality issues drove the result. You are asked to measure the causal impact of receiving negative reviews on a merchant’s coupon repurchase rate. Outline the data you need and the methodology you would use to obtain an unbiased estimate.

Hints

Think about sample-ratio mismatch checks, covariate balance, time windows, difference-in-differences, matching/propensity scores, or holdout experiments.