Katkat
Back

Measure and mitigate notification spam

MetaData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design an experimentation framework for notification systems while accounting for user fatigue, long-term ecosystem health, and signal interference.

The Scenario

You are a Data Scientist at a social platform where users receive push notifications for various activities (likes, comments, suggested groups, etc.). Recently, the product team noticed a decline in long-term retention alongside an increase in notification volume. You suspect that users are experiencing 'notification fatigue,' where excessive signals lead to lower engagement or account deactivation.

The Task

Design a strategy to identify and mitigate notification spam. Your proposal must address the following:

  1. Metrics: Define primary and guardrail metrics to measure notification 'health' vs. 'spam.'
  2. Experimentation: Propose an experimental design to test the impact of reducing notification volume. How would you handle potential 'interference' (where reducing one notification type affects engagement with another)?
  3. Long-term impact: How do you differentiate between a short-term drop in engagement (due to fewer notifications) and a long-term improvement in user ecosystem health?
  4. Mitigation: Describe a framework for deciding when to retire a specific notification type or move it to a 'digest' format.

Constraints

  • Assume you have data on notification delivery, click-through rates, opt-out rates, and 30-day retention.
  • Consider that user 'churn' caused by spam is often silent (users uninstall rather than opt-out).

Hints

Hint 1

Focus on 'incremental' metrics like lift in sessions per notification sent, rather than just raw volume.

Hint 2

Think about using 'Switchback' or 'Cluster-randomized' experiments to mitigate the interference between different notification types within the same user's feed.

Hint 3

Consider utilizing 'User Surplus' or 'Retention adjusted for notification frequency' as a north-star metric to capture the trade-off between engagement and fatigue.

Follow-up questions

  • How would you account for 'novelty effects' where users react positively to a change simply because it is different, not because it is better?
  • If reducing notification volume leads to a short-term drop in Daily Active Users (DAU) but an increase in 90-day retention, how would you present this trade-off to stakeholders?
  • How can you leverage machine learning to personalize notification frequency per user instead of applying a global reduction rule?