Katkat
Back

Determine Key Metrics and Design A/B Test for Ad Ranking

MetaData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Key Metrics and Design A/B Test for Ad Ranking states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

Ads team is replacing a rule-based ad ranking with a new recommendation system; UI and auction rules stay the same.

Question

Which primary and guardrail metrics would you track for this launch and why? How would you design the A/B test so control and treatment populations are truly comparable? Write the formulas you would use to estimate required sample size and test duration. If treatment CTR rises 5%, will advertisers necessarily spend more? Explain the causal path and additional analyses you would run. Below is a chart where treatment CTR is already higher than control during the pre-launch period. What is wrong with this picture and how would you fix the visualization? How would you summarize results, next steps, and risks for senior leadership?

Hints

Think metric hierarchy, randomization, statistical power, business incentives, visualization best-practices, and executive storytelling.