Katkat

Compute duration and stopping rules correctly

InstacartData Scientist · Onsite

Last updated

This question evaluates a data scientist's competency in experimental design and applied statistics, specifically power and sample‑size calculations, accounting for intra‑household clustering and design effects, sequential monitoring for peeking control, and variance‑reduction or stratified randomization to handle day‑of‑week seasonality.

You must pre‑compute how long to run an experiment on same‑day delivery attach rate. Baseline attach rate = 22%, minimum detectable relative lift = 5%, α=0.05 (two‑sided), power=0.8. Daily traffic = 120k eligible checkouts, but only 20% of site traffic is eligible; eligibility and traffic vary by weekday with strong Sunday spikes. a) Calculate required sample size per arm and translate to calendar days accounting for day‑of‑week seasonality and a 10% cluster correlation due to households (design effect). b) Define a peeking‑safe monitoring plan (e.g., group sequential or alpha spending), and show how it changes expected duration. c) If the metric is volatile on Sundays (Miami), propose stratified randomization or CUPED to reduce variance; quantify the variance reduction you expect and its impact on duration.