Katkat
Back

Design a News Aggregation System (Google News-style)

RipplingSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design an end-to-end news aggregation platform that ingests articles at scale, deduplicates syndicated content, clusters articles into evolving story threads, and serves low-latency feeds.

Scenario

Design a large-scale news aggregation platform similar to Google News. The system is responsible for continuously ingesting articles from thousands of global news outlets and RSS feeds, identifying near-duplicate or syndicated content, grouping related coverage into coherent "story clusters", and delivering a personalized, ranked feed of breaking and trending news to users.

Requirements

Functional Requirements

  • Continuous Ingestion: Periodically crawl registered RSS feeds and news sites to extract article content, metadata, and publication timestamps.
  • Deduplication: Detect exact copies as well as near-identical syndicated articles.
  • Story Clustering: Group articles covering the same real-world event into a single story entity, even if they come from different sources and use different wording.
  • Trending & Freshness: Identify breaking news topics and promote rapidly evolving clusters.
  • Feed Serving: Provide an endpoint for users to retrieve top stories by category, with basic personalization (e.g., topic/source preferences).

Non-Functional Requirements

  • Freshness: Newly published articles should be processed, deduplicated, and clustered within 5 minutes of publication.
  • Read Latency: Feed retrieval p99 latency should be under 150 ms.
  • Availability: 99.99% availability for the read path.
  • Scalability: Gracefully handle breaking news traffic spikes where read QPS doubles or triples.

Scale & Assumptions

  • Sources: 50,000 active publishers / RSS feeds.
  • Article Ingestion: 10 million new articles per day (~120 articles/sec average, 500 articles/sec peak).
  • User Base: 50 million Daily Active Users (DAU).
  • Read Traffic: ~25,000 queries per second (QPS) at peak.
  • Active Clustering Window: Cluster articles within a rolling 7-day window; archive older clusters to cold storage.

What Your Design Should Cover

  1. High-Level Architecture: End-to-end data flow from crawler/ingest worker to user-facing API.
  2. Deduplication & Clustering Pipeline: Algorithms and data structures used for near-duplicate detection (e.g., locality-sensitive hashing) and clustering (e.g., incremental vector similarity vs. batch clustering).
  3. Data Model & Storage: Schema design for articles, clusters, and user interactions; choice of databases (document store, relational, vector index, cache).
  4. Feed Generation & Ranking: Read path architecture, caching strategy, and scoring mechanisms for breaking/trending stories.
  5. Trade-offs & Edge Cases: Managing crawler politeness, drift in long-running story clusters, and handling spam or low-quality sources.

Hints

Hint 1

Separate exact duplicate detection (URL normalization and content hashing) from semantic near-duplicate detection using algorithms like SimHash or MinHash.

Hint 2

Avoid running global clustering algorithms across the entire corpus; instead, project incoming articles into an embedding space and perform an incremental nearest-neighbor search against active cluster centroids.

Hint 3

Decouple the heavy write/clustering pipeline from the user read path by pre-computing ranked cluster feeds and caching them in a distributed key-value store.

Follow-up questions

  • How would you prevent cluster drift, where a single long-running story (e.g., an ongoing election) absorbs unrelated political events over time?
  • How would you design crawler politeness and distributed rate limiting to avoid overwhelming smaller publisher websites?
  • What scoring function would you use to balance story freshness, publisher authority, and user engagement when ranking clusters?