Katkat
Back

How to evaluate a similar-listing notifications feature

MetaData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

A Meta Data Scientist analytics-and-experimentation interview question: a US C2C second-hand marketplace is considering opt-in “similar listings” notifications. You must (1) decide whether to build it — hypotheses, opportunity sizing, risks, MVP plan — and (2) design the post-launch measurement: a user-randomized A/B test with a primary/diagnostic/guardrail metric stack, handling of opt-in selection bias, and a clear launch/rollback rule.

Question

You are a Data Scientist on a US C2C marketplace app (like Facebook Marketplace) where users buy and sell second-hand products.

Current product behavior

  • Users browse product listings.
  • If a buyer is interested in a listing, they can click “Send message” to contact the seller.
  • Each message sent counts as one listing interaction.

Proposed feature On a product listing, buyers can opt into reminders/notifications for “similar listings you may like.” When similar products become available, the buyer receives a notification.

Answer the following:

  1. Pre-launch / decision framing. How would you decide whether this feature is a good idea for the product? Cover:
  • The user problem and hypothesis you are testing.
  • What success metrics you would expect to move (and why), and how you would distinguish primary vs. diagnostic vs. guardrail metrics.
  • Key tradeoffs and risks (e.g., notification fatigue, adverse selection, cannibalization of search).
  • What data you would analyze before building to validate demand and size the opportunity (e.g., backtesting against historical logs).
  • What MVP / phased rollout plan you would propose if you were uncertain.
  1. Post-implementation impact evaluation. Assume engineers have shipped the functionality (or it can be enabled for some users). How would you measure its impact and determine whether it is successful? Be specific about:
  • The recommended experiment or causal design, the unit of randomization, control vs. treatment, and duration.
  • Primary success metric(s) vs. secondary/diagnostic metrics vs. guardrail metrics.
  • Key pitfalls (opt-in selection bias, notification fatigue, interference/network effects, seasonality, attribution) and how you would handle them.
  • How you would interpret results and decide to iterate, roll out, or roll back.