Katkat
Back

Measure impact of ads-manager automation feature

RedditData Scientist · Onsite
EasyData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests your ability to design an evaluation framework for a new feature, specifically focusing on causal inference, metric selection, and identifying potential biases in a two-sided marketplace.

Scenario

Reddit is launching a new 'Smart Bidding' automation feature for its Ads Manager. This feature uses machine learning to automatically adjust bids for advertisers to maximize conversions while staying within their budget. You need to design an evaluation plan to determine if this feature actually increases total advertising revenue and advertiser satisfaction.

Task

  1. Propose a primary metric and two secondary metrics to evaluate the feature's success.
  2. Describe how you would set up an experiment (e.g., A/B test) to measure the causal impact of this feature on ad revenue.
  3. Identify two potential biases or 'network effects' that could arise in this experiment and explain how you would mitigate them.

Considerations

  • Ads systems are two-sided marketplaces: changing how one advertiser bids affects the cost and visibility of others.
  • Assume you have access to historical campaign data, user engagement logs, and advertiser churn records.
  • Your answer should address the trade-off between short-term revenue gains and long-term advertiser retention.

Hints

Hint 1

Consider metrics that balance immediate spend (revenue) against the efficiency of that spend for the advertiser (ROAS).

Hint 2

Think about whether randomizing at the individual user level, the advertiser level, or the ad-slot level is most appropriate for a bidding system.

Hint 3

Network effects often occur because one advertiser's increased bid can 'crowd out' another advertiser, potentially skewing control group results.

Follow-up questions

  • How would you measure the 'cannibalization' effect, where automated bids simply shift spend from one type of inventory to another without increasing total budget?
  • If the automated feature only rolls out to a subset of advertisers, how do you ensure the test group is representative of the entire advertiser base?
  • What specific instrumentation or logging would you require from the engineering team to distinguish between 'manual' and 'automated' bidding interventions?