Katkat
Back

Evaluate Factors Before Replacing Recommendation Model

MetaData Scientist · Onsite
HardData ScientistOnsiteMachine Learning

Last updated

Evaluates whether an ads recommendation model should replace an existing production model. Strong answers combine offline validation, online experiments, revenue and advertiser outcomes, user guardrails, conflicting metric interpretation, rollout safety, rollback planning, and executive communication.

Scenario

Facebook Ads team built a new recommendation model and plans to deprecate the old one.

Question

a) What factors must be evaluated before fully replacing the existing model?

b) CTR falls but revenue or other metrics increase—how would you decide which model to ship?

c) Outline the A/B-testing procedure you would follow.

d) How would you visualise and present the final results to the CFO?

Hints

Cover offline evaluation, guardrails, rollback plans, stakeholder dashboards, statistical significance.