Katkat
Back

How to evaluate adding video ads in a game

AmazonData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design an experimentation framework and evaluate the business impact of introducing interstitial video ads in a mobile game, focusing on the trade-off between short-term revenue and long-term user retention.

Scenario

You are a Data Scientist for a popular free-to-play mobile game at Amazon. The product team wants to introduce 30-second 'rewarded video ads'—where players choose to watch an ad in exchange for in-game currency—to increase monetization. You need to design an experiment to determine if this feature should be rolled out globally.

The Task

  1. Define the primary, secondary, and guardrail metrics you would use to evaluate this experiment.
  2. Describe the A/B testing design, including how you would define the randomization unit and the duration of the test.
  3. Explain how you would account for potential 'cannibalization,' where players who watch ads might be less likely to purchase in-game currency through direct payments.

Data Assumptions

  • You have access to granular event logs (player sessions, ad-view events, IAP transactions).
  • You have user-level demographic data and historical engagement metrics (DAU, sessions/day, session duration).
  • Assume a 14-day observation window for the experiment.

Hints

Hint 1

Consider metrics that measure both immediate revenue generation and the long-term health of the user base.

Hint 2

Think about how the randomization unit (e.g., User ID vs. Device ID) impacts the potential for cross-contamination between test and control groups.

Hint 3

Use a causal approach to isolate the impact of ads on IAP (In-App Purchase) revenue rather than just looking at aggregate averages.

Follow-up questions

  • If you observe a significant increase in ad revenue but a slight decrease in D7 retention, how would you decide whether to proceed with the launch?
  • How would you adjust your analysis if the game has a highly skewed distribution of 'whales' (high spenders) versus 'minnows' (non-spenders)?
  • How could you estimate the long-term Customer Lifetime Value (LTV) impact based on only 14 days of experiment data?