Katkat
Back

Design an Experiment to Evaluate New ML Model

MetaData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design an Experiment to Evaluate New ML Model states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

Ads platform wants to validate engineers’ claim that a new ML ranking model outperforms the existing recommender.

Question

How would you design an experiment to evaluate the new ML recommendation system? Which primary and guard-rail metrics would you monitor and why? An A/B test shows a 5% lift in CTR—how do you judge practical significance? CTR doubles for Indian males aged 18-55—what might this indicate and what next steps would you take? If the test yields +5% CTR and +5% revenue, would you roll the model out globally? Explain your decision process.

Hints

Discuss randomization, sample size, heterogeneous effects, business trade-offs, and ethical checks.