Measure Success for a New Chat System
RobloxData Scientist · Technical Screen
Last updated
Build a chat-system launch framework with meaningful communication metrics, reliability and safety guardrails, and experiments that address network effects.
Measure Success for a New Chat System
A social platform is considering a new chat system. Design a measurement framework and an experiment to decide whether the launch is successful. Explain how you would choose a primary metric, guardrails, randomization unit, and launch criterion. The new system's specific changes are not supplied, so begin by clarifying the intended user benefit.
Hints
Hint 1
Messages involve more than one user: A change seen by one participant may change the experience or behavior of another participant assigned to control.
Follow-up questions
- How could total messages increase while the chat experience worsens?
- What would you do if adoption rose but conversation reliability degraded?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.