Choose group-call size cap via experiment
MetaData Scientist · Technical ScreenLast updated
This question evaluates a candidate's competency in experimental design, A/B testing and statistical analysis for product decisions, specifically testing metric selection, reliability and latency guardrails, stratified randomization, interference and clustering effects, heterogeneity detection, and operational stopping rules.
You must decide the maximum participants per Group Call. Propose an experiment to select the cap among {4, 8, 16}. Cover: (a) Design choice (fixed multi-arm vs bandit) under a 4-week deadline; justify exploration vs exploitation trade-offs. (b) Primary metric (e.g., successful starts per eligible user) and reliability/latency guardrails (start success rate, join latency p95/p99, crash rate). (c) Stratified randomization to balance device class, network type, and region; how to handle cluster/household effects. (d) Detect non-monotonic effects (8 may outperform 4 and 16) and heterogeneity by segment (friends vs workgroups); specify an analysis that controls false discovery (e.g., hierarchical modeling or Holm-Bonferroni). (e) Stopping rules, interim looks, and rollback thresholds if tail latency SLOs are breached; include how you would cap exposure to protect reliability while still making a decision.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.