Katkat
Back

Design a news aggregator like Google News

RipplingSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This problem tests your ability to design a distributed system that fetches, normalizes, and ranks news articles from various heterogeneous sources while maintaining high availability and low latency for users.

Problem Statement

Design a system similar to Google News that aggregates articles from thousands of RSS feeds and news sites, personalizes the feed for users, and serves them with minimal latency.

Functional Requirements

  1. Ingestion: The system must fetch new articles from various web sources periodically.
  2. Aggregation/Ranking: Articles must be categorized (e.g., Politics, Tech, Sports) and ranked based on relevance and freshness.
  3. User Feed: Users can request a personalized news feed based on their interests.
  4. Search: Users should be able to search for articles using keywords.

Non-Functional Requirements

  1. Scale: Support 10 million daily active users and process 100,000 new articles per day.
  2. Latency: Feed loading time should be under 200ms.
  3. Freshness: New content should appear in the feed within 5 minutes of publication.

Deliverables

  1. High-level architecture diagram (described in text).
  2. Choice of storage systems for raw articles, processed articles, and user interest profiles.
  3. Strategy for handling high read traffic and spikes in news volume.
  4. API endpoints for feed retrieval and search.

Hints

Hint 1

Consider a decoupled architecture using a message queue (e.g., Kafka) to separate the heavy ingestion/parsing logic from the user-facing read path.

Hint 2

Think about how to use an inverted index (e.g., Elasticsearch) for searching compared to a relational database or NoSQL store for metadata.

Hint 3

Address the 'feed generation' problem: Is it better to pre-compute feeds using a worker pool or compute them on-the-fly when a user requests?

Follow-up questions

  • How would you handle deduplication of news articles that cover the same event from different sources?
  • If a major breaking news event occurs, how would you prevent the system from being overwhelmed by a sudden spike in traffic?
  • How would you evolve the personalization engine to include collaborative filtering rather than just static user interests?