Katkat
Back

Investigate Homepage Experiment Without Control Group: Methods and Metrics

PinterestData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

Evaluates quasi-experimental analysis and product-metric judgment after a homepage launch lacks a randomized control. Strong answers compare matching and propensity weighting, audit experiment hygiene, choose metrics, and diagnose mixed CTR results.

Scenario

Several experimentation and product-metric challenges for a social-media homepage.

Question

An intern launched an experiment without a control group—how can you still estimate treatment impact? Compare matching versus propensity-score weighting and discuss their trade-offs. Review an existing A/B test and list common pitfalls that could bias the results. For a new horizontal home-feed module, what primary metrics would you track to judge success? Post-launch you observe homepage click-through dropping in treatment while DAU and time-spent stay flat—how would you investigate root causes and which user segments would you analyze first?

Hints

Think causal inference, experiment design, diagnosable metrics hierarchy, segmentation by device, geography, tenure, power-users vs casual, etc.