Katkat
Back

Measure Local News Launch Success

NextdoorData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your ability to design an experimentation framework for a localized product launch and identify potential confounding factors and interference effects.

Scenario

Nextdoor is launching a new "Local News" feature that allows local journalists to post verified updates directly to neighborhood feeds. You are tasked with determining if this feature increases user engagement within the target neighborhoods.

The Task

Design an experiment to evaluate the success of the Local News launch. Address the following components:

  1. Metric Definition: Propose one primary metric to measure success, one guardrail metric to ensure no negative impact on user experience, and explain the rationale for each.
  2. Randomization Strategy: Would you randomize at the user level or the neighborhood level? Justify your choice by discussing the risk of network interference (spillover effects).
  3. Causal Challenges: How would you account for selection bias if the launch is only rolled out to neighborhoods that "opt-in" to the pilot program?
  4. Sample Size & Duration: Briefly describe how you would determine the necessary duration of the experiment to ensure statistical significance, given the seasonality of local news consumption.

Hints

Hint 1

Consider how a user reading news in one neighborhood might influence or be influenced by residents in adjacent neighborhoods.

Hint 2

Think about the difference between 'intent-to-treat' and the actual impact when neighborhoods self-select into a pilot program.

Hint 3

Guardrail metrics should focus on negative outcomes, such as high churn rates or increased 'hide post' actions.

Follow-up questions

  • How would you adjust your analysis if the Local News feature is a shared resource that journalists can cross-post to multiple adjacent neighborhoods?
  • If the experiment results show a significant increase in engagement, how do you verify that this wasn't caused by a 'novelty effect' that wears off after a few weeks?
  • How would you approach the analysis if you were restricted to a quasi-experimental design (e.g., Difference-in-Differences) because a randomized controlled trial was deemed too operationally complex?