Katkat
Back

Formulate hypotheses and metrics for video-pin ramp

UpstartData Scientist · HR Screen
HardData ScientistHR ScreenAnalytics & Experimentation

Last updated

This question evaluates experiment-design and data-science competencies including hypothesis specification, metric choice and guardrails, segmentation, interference handling, rollout planning, power/MDE considerations, and data-quality monitoring, and it falls under the Analytics & Experimentation domain.

Pinterest plans to increase the proportion of video pins in the home feed to boost engagement. Design the A/B test end-to-end. Tasks:

  • State precise hypotheses: H0 and H1 for the primary metric; list at least two alternative hypotheses capturing possible trade-offs (e.g., engagement up but quality down), and specify whether they are one- or two-sided.
  • Define the experiment unit, randomization, exposure, and target population (e.g., all users vs. US new users as defined by signup within 30 days of action date). Address user-to-user interference and content supply constraints.
  • Choose a single primary metric and 2–4 secondary metrics plus guardrails (e.g., hide rate, complaint rate). Justify each metric’s sensitivity and directionality.
  • Specify segmentation cuts (e.g., pin_format, device, new vs. existing users) and how you will handle heterogeneous effects without p-hacking.
  • Detail the rollout plan (holdout size, ramp schedule, duration), data quality checks (event logging completeness, attribution), and stopping rules (early stop for harm, futility, or success).