Katkat
Back

Design a Deduplicated Personalized News Feed

RipplingSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a news aggregation and recommendation system that ingests multi-publisher articles, clusters coverage of identical real-world events, and serves a personalized, deduplicated feed with low latency.

Scenario

Modern news aggregators ingest content from thousands of publishers. Often, dozens of outlets report on the exact same breaking news event simultaneously. Showing multiple articles covering the same development degrades user experience.

Design a personalized news feed system that automatically identifies and groups coverage of the same underlying event into unified "story clusters" while maintaining source diversity, ranking relevance, and freshness.

Requirements

Functional Requirements

  • Ingestion & Deduplication: Ingest articles continuously, detect near-duplicate coverage, and group articles into event-level clusters (story cards).
  • Source Diversity: Within each cluster, maintain links to distinct publisher perspectives without cluttering the main feed.
  • Personalization: Rank clusters in a user's feed based on topic preferences, reading history, and event importance.
  • Read API: Provide an API to fetch the paginated personalized feed and an API to view all publisher sources for a given cluster.

Non-Functional Requirements

  • Low Latency: Feed retrieval latency under 200 ms at p95.
  • Freshness: Breaking news articles should be clustered and eligible for feed inclusion within 2 minutes of ingestion.
  • Scalability: Must support high read traffic and dynamic personalization without redundant re-clustering.

Scale & Traffic Estimates

  • Daily Active Users (DAU): 50 million.
  • Read Load: Average 5 feed requests per user per day (~3,000 QPS average, 10,000 QPS peak).
  • Ingestion Load: ~100,000 new articles ingested per day across all monitored outlets (~10 articles/sec peak).
  • Storage: Metadata and text embeddings for 100,000 articles daily (~500 MB to 1 GB/day, retained for active clustering across a rolling 7-day window).

Key Areas to Cover

  1. Data Model & Clustering Pipeline: How to represent articles and clusters. Algorithms or approaches for near-duplicate and semantic clustering (e.g., MinHash/SimHash, text embeddings with sliding time-window vector search).
  2. High-Level Architecture: Ingestion pipeline, streaming/asynchronous clustering worker, vector/metadata storage, and feed generation service.
  3. Feed Personalization & Ranking: Combining user preference vectors with cluster-level features (freshness, publisher credibility, engagement, topic tags).
  4. Trade-offs & Deep Dives: Push vs. pull model for feed delivery, handling evolving/developing stories, and selecting representative canonical titles and images for a cluster.

Hints

Hint 1

Bound clustering candidate searches by both semantic similarity and a rolling publication time window to prevent conflating distinct events separated by months.

Hint 2

Decouple the write-heavy clustering pipeline from the read-heavy ranking path by persisting pre-formed clusters with aggregated features.

Hint 3

Consider multi-stage ranking (candidate generation of top clusters followed by lightweight re-ranking using user interest features) to meet the 200 ms latency target.

Follow-up questions

  • How does your clustering mechanism update when a developing story evolves significantly over 24 hours?
  • How would you ensure publisher diversity within the feed so a single dominant outlet does not monopolize canonical cluster representations?
  • What changes would you make if you needed to immediately push breaking news alerts for high-priority clusters to millions of users?