Katkat
Back

How to evaluate a new Carousel feature

PinterestData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design an A/B test for a feed-based UI feature, focusing on metric selection, counter-metrics, and the identification of potential cannibalization effects.

Scenario

Pinterest is introducing a new 'Carousel' module within the user home feed. This module allows users to horizontally swipe through a set of related Pins instead of scrolling past them vertically. You are the lead Data Scientist tasked with evaluating the impact of this feature through an A/B test.

Task

  1. Define your primary 'North Star' metric for this experiment and explain why it is the best indicator of success.
  2. List at least two guardrail (counter) metrics to ensure the Carousel does not negatively impact the overall ecosystem.
  3. How would you determine if the Carousel is increasing engagement or simply cannibalizing clicks from other parts of the feed?

Constraints

  • Assume you have access to standard event logging (impressions, clicks, saves, session duration).
  • The experiment will run for 14 days.
  • Consider both short-term engagement and long-term user experience.

Hints

Hint 1

Consider the difference between user-level engagement and individual element-level performance.

Hint 2

Think about what 'cannibalization' looks like in terms of total session volume versus attribution to specific modules.

Hint 3

Guardrail metrics should focus on core business health, such as latency or negative user feedback, to ensure the new UI doesn't degrade the core experience.

Follow-up questions

  • How would you segment the results to see if the Carousel performs differently for new users versus power users?
  • If the experiment shows a lift in total clicks but a decrease in total session duration, how would you interpret these conflicting signals?
  • What statistical methods would you use to account for the 'novelty effect' where users click on the new feature simply because it is new?