Katkat

Design an A/B test for pinned-unread feature

MetaData Scientist · Technical Screen

Last updated

This question evaluates proficiency in experiment design and causal inference for product analytics, including choosing the randomization unit, defining eligibility and exposure, selecting primary and guardrail metrics, performing power/MDE and sample-size analysis, and addressing identity cross-contamination and network effects.

Design a rigorous experiment to evaluate the pinned-unread chat feature. Cover: (a) the unit of randomization and assignment (user_id vs person_id given some persons have multiple user_ids) and how you’ll prevent cross-contamination; (b) eligibility criteria and pre-exposure; (c) primary and secondary metrics (e.g., reply rate within 24h, messages read per day, time-to-first-response, net unread count change), plus guardrails (crash rate, client CPU/memory, API latency); (d) power/MDE and sample size approach, accounting for heavy-tailed engagement; (e) test duration and seasonality controls; (f) novelty/learning effects and ramp plan; (g) handling network effects (group chats) and SUTVA risks; (h) bucketing for users who frequently switch accounts; (i) instrumentation/data quality checks (duplication, clock skew, delayed reads); and (j) a pre-registered analysis plan, including variance reduction (e.g., CUPED) and how you’ll interpret conflicting metric movements.