Design and assess a video-pin increase experiment

Question

Pinterest wants to increase the share of video pins surfaced in the Home Feed (e.g., raising video share from a ~30% baseline toward a 45% target, an increase of roughly +10 to +15 percentage points) to boost engagement. Design a rigorous evaluation, then interpret provided results and make a recommendation.

A) Experiment design

1. Unit of randomization & exposure control. Specify the unit of randomization (user-level vs. session-level) and justify it given network/content-supply interference and feed-ranking spillovers. State how you would cap per-user exposure to the new mix (e.g., from a 30% baseline video share to a 45% target) and how you would ramp.
2. Metrics. State a clear primary hypothesis plus at least two alternative hypotheses (e.g., substitution effects, novelty effects, reliability cost). Define a primary success metric and guardrails with exact formulas (e.g., saves_per_impression, clicks_per_impression, time_spent_per_user_day, complaint_rate = complaints / impressions, session_end_rate, creator churn/follows, crash rate, bandwidth cost per user). Justify each and state win/loss directions.
3. Power, duration & sequential analysis. Outline your power / MDE and duration assumptions (alpha, two-sided test, allocation, variance source), and how you will handle sequential looks or peeking (e.g., group sequential boundaries or CUPED for variance reduction). Include an A/A check, a manipulation check (did video share actually move?), and a novelty/fatigue plan (minimum run and long-term holdout).
4. Quasi-experiment. If an RCT is infeasible, propose a credible quasi-experiment (e.g., staggered-rollout difference-in-differences with user fixed effects + inverse-propensity weighting, or synthetic control). List identifying assumptions, diagnostics, and sensitivity checks.

B) Interpret the readout(s) — assess statistical significance, directionality, and practical significance; call out red flags (SRM, multiple testing, heterogeneous effects, cross-unit metrics). Then recommend whether to ship, ramp/iterate, or stop, justify using the metrics, multiple-testing/guardrail considerations, and potential mitigations (e.g., cap video share for sensitive cohorts, rank-quality filters), and state what additional data or follow-up analysis you would run before a full rollout.

Readout 1 — 14 days, user-level randomization, robust SEs (N ≈ 2.0M users)

Readout 2 — 14 days (alternate run, with absolute lifts and CIs)

In both cases: does the CTR win offset the negative guardrail (retention/complaints in Readout 1, crash rate in Readout 2)? If not, what mitigations or follow-ups would you require before full rollout?