Katkat
Back

Design experiment for ads in chat with budgets

RedditData Scientist · Onsite
EasyData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to design an A/B test for a new ad placement in a chat feature while accounting for budget constraints and potential network effects.

Scenario

Reddit is considering introducing native ads directly within private chat threads to increase monetization. You are tasked with designing an experiment to determine if this change increases total platform revenue without significantly degrading user engagement. Because advertisers have strict daily budget caps, showing ads in chat might simply 'cannibalize' impressions from other surfaces like the Feed, rather than creating incremental value.

Task

Design an experiment to measure the causal impact of chat ads on total platform revenue. Your response should address the following:

  1. Randomization Strategy: How will you assign users/units to treatment and control? Explain why you chose this unit of randomization to minimize interference.
  2. Metrics: Define one primary metric for business impact, one guardrail metric for user experience, and one metric to monitor for cannibalization.
  3. Budget Constraints: How will you account for the fact that advertisers have fixed budgets, which might lead to artificial displacement effects?

Example

Consider a user who has a daily ad budget of $5. If they see ads in chat, they might exhaust their budget earlier in the day, causing them to see fewer ads in the Feed. A naive test might show an increase in chat revenue but an equal or greater decrease in Feed revenue, resulting in zero net lift.

Hints

Hint 1

Consider whether you should randomize at the user level or the cluster/network level to account for the social nature of chat.

Hint 2

Think about the difference between 'net platform revenue' and 'revenue per surface' when assessing cannibalization.

Hint 3

To handle budget constraints, consider using an 'intent-to-treat' design or a controlled experiment where budget allocation is randomized alongside ad placement.

Follow-up questions

  • How would you adjust your analysis if you suspected that chat ads caused a decrease in the number of messages sent per user?
  • If the experiment results show a lift in revenue but a slight decrease in retention, what statistical framework would you use to make a 'ship vs. no-ship' decision?
  • How would you conduct a power analysis if the chat-ads interaction rate is expected to be significantly lower than feed-ads?