Evaluate Model-Targeted Subscription Notifications
DiscordData Scientist · Technical Screen
Last updated
Evaluate subscription-targeting notifications with incrementality metrics, A/A preparation, flat-result diagnosis, and alternative causal designs.
Evaluate Model-Targeted Subscription Notifications
A classification model identifies users likely to subscribe, and the product team plans to send those users a notification. Design a measurement and experimentation plan. Include checks before the experiment, how to interpret a flat result, and alternatives to a conventional user-level A/B test with their trade-offs. Explain how the plan separates model prediction quality from the incremental effect of sending notifications.
Hints
Hint 1
A likely subscriber may subscribe anyway: High predicted conversion does not establish that the notification changes the user’s decision.
Follow-up questions
- How would you decide whether to target likely subscribers or users most persuadable by a notification?
- What evidence can an offline classifier evaluation provide before any notification is sent?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.