Katkat
Back

Evaluate New Ad Model with A/B Testing Experiment

MetaData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Evaluate New Ad Model with A/B Testing Experiment states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

As part of the Ads team you have trained a new ad-recommendation model and need to decide if it should replace the current model.

Question

Design an experiment to evaluate the new recommendation model against the incumbent.

What primary and guardrail metrics would you track? How would these differ when presenting results to the CFO versus the CGO (growth)?

State your final recommendation framework for launch or rollback.

Hints

Cover experiment design (A/B, traffic split, duration), metrics (revenue, CTR, ROI, user retention), statistical significance and trade-offs for finance vs growth.