Katkat
Back

Determine High-Quality Notifications with CTR Analysis

MetaData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine High-Quality Notifications with CTR Analysis states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

Mobile app sends push notifications and wants to ensure only high-quality notifications are shipped.

Question

Propose and justify a metric that captures high-quality notifications. Given historical click-through-rate (CTR) data, how would you determine whether the current CTR is good or bad? Design an A/B test for a new notification strategy; explain experiment setup, success metrics, and how you would analyze the results in depth.

Hints

Cover metric formulation, baselines, segments, statistical power, validity threats, and post-test analysis.