Design Product Notification and Autocomplete Experiments

You are interviewing for a Product Data Scientist role at Discord. Answer both product experimentation case studies below.

Case 1: Subscription notification experiment

You are the data scientist for a product team that wants to send a notification to users who are highly likely to sign up for a paid subscription service.

1. How would you define a "power user" or "high-intent user" for this use case?
2. How would you design an experiment to test whether sending the notification is beneficial?
3. If the experiment is successful, what should the team explore next?

In your answer, consider targeting criteria, success metrics, guardrail metrics, experiment unit, randomization, sample-size planning, selection bias, heterogeneous treatment effects, and long-term user experience.

Case 2: Mobile autocomplete experiment

The mobile team wants to A/B test an autocomplete feature on the "Add Friend" page. Today, if a user enters a username that does not exist, the product returns a null result. The proposed feature would help users complete or correct the username through autocomplete suggestions.

1. How would you run this experiment?
2. What metrics would you use to determine whether the experiment is successful?

In your answer, consider the user journey, primary and secondary metrics, guardrails, randomization level, logging requirements, and possible risks such as privacy concerns, incorrect friend suggestions, or changes in spam behavior.