Katkat
Back

Evaluate Instagram's Short-Video Recommender System Success

MetaData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

Evaluates success metrics and experimentation for Instagram's short-video recommender feed. Strong answers choose a durable user-value metric, understand heavy-tailed engagement distributions, handle metric trade-offs, and outline a safe A/B test with retention, safety, creator, and app-level guardrails.

Scenario

Instagram is launching a short-video recommender feed.

Question

a) Which single metric would you use to evaluate the recommendation system’s success and why?

b) Sketch or describe the expected distribution of that metric, labelling median, mode, and 95th percentile.

c) Metric A rises while Metric B falls—what do you do?

d) List the end-to-end A/B-testing steps for this launch.

Hints

Discuss engagement vs retention, heavy-tailed distributions, guardrail metrics, power calculation and success criteria.