Katkat
Back

Design and Interpret a Video Pin Experiment

PinterestData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design a balanced A/B experiment for a two-sided marketplace (creators and viewers) and synthesize conflicting metrics into a launch decision.

The Scenario

You are a Data Scientist at a platform where users create and view video Pins. The Product team wants to launch a new 'Simplified Video Editor' to lower the barrier to entry for content creation. You need to design an A/B experiment to determine if this feature should be launched.

Your Task

  1. Experiment Design: Define your randomization unit and how you will handle potential spillover effects between creators and viewers.
  2. Metric Framework: Propose a set of primary metrics (for creation volume), secondary metrics (for viewer engagement), and guardrail metrics (for content quality/safety).
  3. Trade-off Analysis: Imagine the experiment results show a 5% increase in total video uploads (primary), a 2% increase in time spent watching videos (secondary), but a 15% relative increase in the 'report-rate' (user flags for spam or low-quality content) on newly created videos.

Constraints

  • The platform relies on high-quality organic content to maintain user trust.
  • You have access to user-level metadata and engagement logs.
  • Assume a 14-day experiment duration.

Provide a framework for how you would analyze these results and what your final recommendation would be. If the guardrail metric (report-rate) is statistically significant but the growth metrics are also positive, how do you decide to launch or iterate?

Hints

Hint 1

Consider if randomization should occur at the user level or the content level, and whether the 'Simplified Video Editor' influences existing content quality.

Hint 2

Think about the difference between 'upload volume' and 'upload quality'; a simple editor might make it easier to upload low-effort, low-value content.

Hint 3

Address the trade-off by proposing a segmentation analysis—does the report-rate spike only affect new, less-experienced creators or all users?

Follow-up questions

  • How would you account for 'novelty effect' where users try the new tool once but don't return to use it long-term?
  • If the report-rate guardrail is breached, what specific technical or product changes would you propose to mitigate spam before re-testing?
  • How would you calculate the 'net impact' on the ecosystem if creator creation increases but viewer churn also increases due to lower quality content?