Run org-safe online experiment for recommender
DropboxData Scientist · Technical ScreenLast updated
This question evaluates a data scientist's competency in online experimentation design, causal inference, clustered randomization, metric definition (primary, secondary, guardrails), sample size/MDE computation, variance reduction and sequential monitoring, instrumentation and privacy-aware logging.
Propose an online experimentation plan to evaluate the file recommender in production across multiple organizations where collaborators can influence one another. Specify: primary metrics (e.g., file open-through-rate, time-to-open), secondary/business metrics (productivity proxies), and guardrails (latency, error rate, privacy incidents, access denials). Choose the unit of randomization (org-, team-, or user-level) and justify to minimize spillover; describe bucketing, stickiness, and holdouts. Compute required sample size and MDE with clustering/ICC assumptions; select variance reduction (CUPED/stratification) and sequential monitoring approach with alpha spending. Detail ramp schedule, novelty and carryover controls, and interference detection. Define logging needed to reconstruct exposure and attribution, plus a difference-in-differences fallback if only partial randomization is possible. Explain stop/ship criteria and how to guard against Simpson’s paradox across tenants and roles.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.