Katkat
Back

How would you evaluate a carousel launch?

PinterestData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design a comprehensive evaluation framework for a product feature launch, specifically focusing on metric selection, causal impact measurement, and addressing potential cannibalization effects.

Scenario

Pinterest is launching a 'Carousel' feature on the home feed, which allows users to swipe through multiple images within a single pin-like card. As a Data Scientist, you are responsible for defining the success of this launch.

Task

Design an evaluation plan for this feature. Your response should cover the following areas:

  1. Metric Framework: Define one primary metric, two secondary metrics, and at least one guardrail metric to monitor for negative externalities.
  2. Experimental Design: Describe how you would set up an A/B test to measure the impact of this feature. How do you handle potential user interaction changes (e.g., users spending more time on the feed but clicking fewer pins)?
  3. Trade-off Analysis: If the Carousel increases 'Time Spent' but decreases 'CTR (Click-Through Rate)' on individual pins, how would you decide whether the launch is a success?
  4. Diagnostic Analysis: How would you differentiate between a short-term 'novelty effect' and long-term behavioral change?

Hints

Hint 1

Consider how the carousel changes the funnel from 'viewing an ad/pin' to 'interacting with content within the feed'.

Hint 2

Think about the impact on downstream revenue if the carousel surface competes for the same screen real estate as individual promoted pins.

Hint 3

Address the distinction between session-level metrics and user-level metrics to control for variance in user activity.

Follow-up questions

  • How would you account for network effects or content creator behavior changes in this experiment?
  • If the experiment results are statistically significant but the effect size is very small, what secondary data points would you look for to justify a full rollout?
  • How would you measure the 'cannibalization' effect on other surfaces if users spend significantly more time in the carousel?