Katkat
Back

Measure Relevant Feed Success

LinkedInData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to define North Star metrics for a social feed and design an experimentation framework that accounts for long-term ecosystem health and cross-side network effects.

Scenario

You are a Data Scientist for a professional networking feed. The product team wants to increase 'Meaningful Professional Interactions' (MPIs) on the feed. They propose a new ranking algorithm that prioritizes content based on predicted 'comment depth' rather than simple 'likes.'

The Task

  1. Metric Definition: Define a primary metric for this experiment that balances short-term engagement (e.g., clicks, likes) with long-term ecosystem value. How do you distinguish between 'vanity' engagement and 'high-quality' engagement?
  2. A/B Test Design: Describe how you would set up an A/B test for this ranking change. What is your randomization unit, and why? How do you account for network effects, where the content creators (posters) and content consumers (viewers) are in the same population?
  3. Evaluation: Suppose the test shows a 5% increase in total comments but a 2% decrease in total 'likes' and a 1% decline in 'new post creation.' How do you reconcile these results to decide whether to launch the feature?
  4. Counter-metrics: Identify at least two counter-metrics that could signal negative externalities for the platform, even if the primary metric is positive.

Hints

Hint 1

Consider the difference between passive consumption and active contribution when defining your metric.

Hint 2

Think about how the randomization unit choice impacts the spillover effect in a two-sided marketplace.

Hint 3

Use a framework like the HEART framework or a cost-benefit analysis approach to weigh the trade-offs between engagement and retention.

Follow-up questions

  • How would you measure the 'fairness' of the new ranking algorithm to ensure it doesn't disproportionately disadvantage niche professional content creators?
  • If this were a multi-page platform, how would you ensure the feed changes aren't simply cannibalizing traffic from other parts of the site?
  • How would you approach a long-term holdback (or 'switchback') experiment to measure the permanent impact of this change on user behavior?