Katkat
Back

How would you evaluate adding video ads?

AmazonData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question evaluates a data scientist's competencies in experimental design, causal inference, metrics selection, and product monetization analytics within the Analytics & Experimentation domain, focusing on measuring the impact of video ads on revenue, engagement, retention, and lifetime value.

You are a data scientist for a free-to-play mobile game. The product team wants to introduce video ads (e.g., rewarded videos and/or interstitial video ads).

Design an evaluation plan to determine whether adding video ads is a good idea.

Your answer should cover:

  1. Success metrics and guardrail metrics (include tradeoffs: short-term revenue vs. retention/LTV vs. engagement).
  2. How you would design an experiment (unit of randomization, eligibility, ad frequency/capping, ramp-up, stratification).
  3. How you would estimate incremental impact on revenue and user experience (e.g., ad revenue offset by changes in IAP purchases, session length, churn).
  4. Key threats to validity (novelty effects, selection bias if ads are opt-in, interference/network effects, bots/ad fraud) and how you would mitigate them.
  5. How you would decide ship / don’t ship, including how you’d think about power/MDE and long-term effects.