Evaluate Impact of New Roblox Homepage Tab
RobloxData Scientist · Technical Screen
Last updated
Roblox data scientist experimentation prompt on evaluating a new homepage tab, including hypotheses, engagement funnel metrics, guardrails, power, duration, segmentation, cannibalization, and rollout criteria.
Scenario
Roblox plans to replace an existing homepage tab with a new one and needs to measure the business impact of this change.
Question
If Roblox replaces an existing homepage tab with a new one, how would you evaluate its impact? Which primary and guardrail metrics would you track, and what hypotheses would you state? Design an experiment or analysis plan outlining sample size, duration, segmentation, and success criteria.
Hints
Think engagement funnel (CTR, session length, retention), controlled A/B test design, power analysis, guardrails, and potential confounders.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.