Katkat
Back

How to evaluate new listing notifications?

MetaData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your ability to design an A/B test for a marketplace notification feature, focusing on balancing buyer engagement with potential cannibalization and ecosystem health metrics.

The Scenario

Meta is introducing a new "Instant Alert" feature for Marketplace. When a new item is posted in a category a user frequently browses, we send a push notification to that user. The goal is to increase the speed of transactions and buyer engagement.

The Task

You are the lead Data Scientist for this feature. Design an experiment to evaluate the impact of these notifications. Specifically, address the following:

  1. Metric Selection: Define the Primary metric, Secondary metrics, and Guardrail metrics you would monitor.
  2. Experiment Design: How would you randomize users to ensure the results are unbiased? Would you use user-level, session-level, or another form of randomization? Explain the rationale regarding network effects.
  3. Causal Interpretation: Notifications might increase short-term engagement but could lead to "notification fatigue" or cannibalize organic search behavior. How do you distinguish between these effects?

Constraints

  • Assume a large, active user base with millions of daily active buyers and sellers.
  • You must account for the two-sided nature of the marketplace (buyers vs. sellers).

Hints

Hint 1

Consider metrics that capture both the immediate conversion (success) and the long-term health of the user's notification settings (sustainability).

Hint 2

Think about whether a notification sent to Buyer A affects Buyer B or the Seller, and how that impacts your choice of randomization unit.

Hint 3

Address the risk of 'cannibalization' by comparing total platform engagement or conversion rates rather than just the performance of the notification channel alone.

Follow-up questions

  • How would you handle the trade-off if the primary metric shows a significant increase, but the guardrail metric regarding 'user block rates' for notifications also increases significantly?
  • If the treatment effect is positive for power users but negative for casual browsers, how would you report your findings to the product team?
  • How would you adjust your experimental design if you suspect the 'Instant Alert' feature causes sellers to lower their prices, thereby creating a marketplace-wide shift rather than a user-specific one?