Katkat
Back

Evaluate Dating App Product Changes

GrindrData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to design an A/B test for a two-sided marketplace, focusing on network effects, cannibalization of metrics, and the selection of hold-out groups to measure long-term user behavior.

Scenario

Grindr is considering a product change to its 'Explore' feature: instead of showing users based strictly on proximity, the algorithm will now prioritize users who are 'highly active' across the entire platform, regardless of distance. Product leadership believes this will increase overall platform engagement time. However, there is a concern that this change may decrease the frequency of local meetups or lead to user frustration for those specifically seeking local connections.

Task

  1. Define primary, secondary, and guardrail metrics to evaluate this change. Explain why each is relevant to a two-sided marketplace.
  2. Design an experimentation strategy. How would you determine the randomization unit (e.g., user-level, cluster/geographic-level) to avoid interference between the treatment and control groups?
  3. Propose a method to differentiate between a short-term 'novelty effect' and a long-term change in platform utility.

Constraints

  • Assume the platform has millions of daily active users.
  • Consider the 'network effect': a change in the behavior of one user (the 'seller' of attention) directly impacts the experience of another (the 'buyer' of attention).
  • Your analysis must account for the difference between 'local' engagement and 'global' platform engagement.

Hints

Hint 1

Consider the 'SUTVA' (Stable Unit Treatment Value Assumption) violation; if a user in the treatment group starts messaging more, how does that affect a control group user who receives their message?

Hint 2

Think about the difference between measuring user-level metrics versus marketplace-level metrics like 'total successful connections' or 'messages exchanged per user.'

Hint 3

For long-term effects, consider implementing a 'switchback' design or a persistent hold-out group rather than a standard user-level randomization.

Follow-up questions

  • How would you account for the fact that a decrease in local match frequency might actually increase long-term user churn?
  • If the experiment shows a lift in total messages sent but a drop in 'offline meetups,' how would you reconcile these conflicting signals?
  • How would you handle cross-platform interference if users have high mobility between different geographic regions?