Katkat
Back

Evaluate Carousel and Billboard Lift

PinterestData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design an A/B test for multi-format advertising, specifically handling cannibalization and cross-format lift measurement.

Scenario

Pinterest is introducing a new 'Billboard' ad unit that appears at the top of a user's feed, while existing 'Carousel' ads continue to appear mid-feed. The product team wants to know if the Billboard drives incremental value or if it simply cannibalizes clicks from Carousel ads. You are tasked with designing an experiment to measure the total impact of adding Billboard ads to the platform.

Task

  1. Define the primary metric you would use to determine if the Billboard ad unit is successful, and explain why this metric accounts for potential cannibalization.
  2. Design the randomization strategy for this experiment. Would you randomize at the user level, device level, or cluster/geo level? Justify your choice regarding network effects or interference.
  3. How would you perform a diagnostic analysis if you observe that the Billboard unit shows a high click-through rate (CTR), but the overall platform engagement time decreases?

Constraints

  • Assume a user can see both Carousel and Billboard ads within the same session.
  • You have access to user-level logs of impressions, clicks, and total time spent on the platform.
  • Consider the trade-off between ad revenue and long-term user retention.

Hints

Hint 1

Consider defining a 'Total Ad Revenue per User' metric that aggregates performance across both ad formats to detect net gains versus substitution.

Hint 2

Think about whether user-level randomization is sufficient, or if there is a risk of a 'satiation effect' where a user's total propensity to click ads is limited regardless of placement.

Hint 3

To diagnose the decrease in time spent, analyze the interaction between the ad format and content consumption latency—does the Billboard push organic content too far down the screen?

Follow-up questions

  • How would you adjust your measurement if you suspect that the presence of a Billboard ad influences a user's likelihood to churn in the long term?
  • If the experiment results show a significant lift in Billboard clicks but a neutral impact on total revenue, what technical steps would you take to investigate potential reporting biases?
  • How would you account for 'novelty effects' where users interact with the new Billboard format purely because it is new, leading to a decay in performance after the first two weeks?