Katkat
Back

Estimate Incremental Search-Ad Revenue Without an A/B Test

RedditData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

Design an evaluation framework to estimate the incremental search-ad revenue of a small-scale manual pilot, incorporating variance reduction, inference under heavy-tailed revenue, and ecosystem cannibalization.

Scenario

Reddit's Search Monetization team recently tested a new high-intent search ad format. Due to infrastructure constraints, the pilot could not run through the standard experimentation platform. Instead, the engineering team manually enrolled 100 randomly selected eligible users into the new treatment experience for 14 days, leaving the remaining eligible users in the standard experience.

You have been asked to lead the quantitative evaluation of this pilot to determine whether the format truly generated incremental search ad revenue and whether it should be scaled to general availability.

Available Data

For all eligible users (both the 100 treated users and the untreated pool):

  • user_id: Unique identifier.
  • treatment_flag: 1 if assigned to the manual pilot, 0 otherwise.
  • pre_search_rev: Total search ad revenue generated by the user in the 14 days prior to the pilot.
  • post_search_rev: Total search ad revenue generated by the user during the 14 days of the pilot.
  • pre_feed_rev, post_feed_rev: Revenue generated from standard feed ads before and during the pilot.
  • pre_searches, post_searches: Total search queries conducted before and during the pilot.

Questions to Answer

  1. Estimand and Identification: Formulate the primary causal estimand. Define whether you are estimating the Intent-to-Treat (ITT) or Average Treatment Effect on the Treated (ATT), and specify the key identification assumptions required given the manual deployment.
  2. Estimation and Variance Reduction: Ad revenue is heavily skewed and zero-inflated, and your treatment cohort has only 100 users. Describe how you would construct your control group, specify the statistical model (e.g., ANCOVA / CUPED vs. difference-in-means), and calculate uncertainty/confidence intervals robust to small-sample skewness.
  3. Cannibalization and Secondary Metrics: How will you detect whether search ad gains came at the expense of feed ad revenue or organic user search frequency? What composite metric would you monitor?
  4. Rollout Decision and Auction Dynamics: Even if the pilot shows statistically significant incremental revenue, why might scaling this feature to 100% of Reddit searchers fail to deliver 100x the pilot's gains? Detail the auction and budget constraints you would flag to leadership.

Hints

Hint 1

Formulate an ANCOVA (CUPED-style) regression using the 14-day pre-period revenue to drastically reduce standard errors given the tiny treatment sample size.

Hint 2

Because revenue distributions are right-skewed with heavy tails and n=100 is small, standard asymptotic normality (z-tests) may fail; consider studentized bootstrap or permutation tests.

Hint 3

Consider ad marketplace mechanics: advertisers have fixed daily budgets, and expanding impressions globally increases supply, which can suppress clearing prices (eCPMs).

Follow-up questions

  • If some users in the treatment group never performed a search during the 14-day window, how does that affect ITT vs. CACE (Complier Average Causal Effect), and which should inform the business decision?
  • How would you select the control group from the broader pool of eligible users to ensure balance, and how would you check for pre-treatment parallel trends?
  • Suppose feed ad revenue dropped by 50 cents for every 1 dollar gained in search ads. How would you construct a decision threshold for leadership to approve full rollout?