Katkat
Back

How would you evaluate emoji reactions launch?

MetaData Scientist · Onsite
EasyData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to design an experimentation framework for a new feature in a social product, specifically focusing on metric selection, identifying interference, and isolating the causal impact of a feature rollout.

Scenario

Meta is planning to launch a new feature: 'Emoji Reactions' for direct messages. Users can long-press a message to add a quick reaction emoji (like a heart or thumbs-up) instead of sending a new text message. You are the lead Data Scientist for the Messaging product team.

Task

  1. Define the key metrics you would use to measure the success of this launch. Distinguish between guardrail metrics, primary metrics, and secondary metrics.
  2. How would you design an A/B test for this feature, considering the risk of network effects (where the behavior of users in the treatment group affects those in the control group)?
  3. Suppose the feature shows a statistically significant increase in emoji reactions but a slight decrease in total message volume. How would you interpret these results and what further analysis would you perform to decide whether to launch the feature?

Constraints & Considerations

  • Explain how you would account for potential cannibalization of traditional text-based replies.
  • Assume you have access to logs of message types, timestamps, and user IDs.
  • Focus on both user engagement and ecosystem health.

Hints

Hint 1

Consider the difference between 'user-level' randomization and 'cluster-level' randomization to handle network interference in messaging networks.

Hint 2

Think about the funnel: does a reaction replace a text response, or does it encourage more conversation overall?

Hint 3

A guardrail metric should monitor for negative unintended consequences, such as an increase in reported spam or a decrease in meaningful text-based communication.

Follow-up questions

  • If we cannot perform a perfect A/B test due to engineering constraints, what quasi-experimental methods (e.g., Difference-in-Differences, Synthetic Control) could we use to estimate the impact?
  • How would you measure the long-term 'stickiness' of the emoji reaction feature compared to short-term novelty effects?
  • If you observe that emoji reactions increase total time spent in the app but decrease the 'depth' of conversations, how would you advise the product manager?