Katkat
Back

Design personalized news aggregation service

RipplingSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a scalable news aggregation system that balances real-time content ingestion with personalized ranking for a large user base.

Problem: Design a Personalized News Aggregator

Design a system that aggregates news articles from thousands of RSS feeds and news APIs, and delivers a personalized feed to millions of daily active users. The system should present users with a list of articles ranked based on their interests and reading history.

Functional Requirements

  1. Ingestion: Fetch and normalize content from external sources periodically.
  2. Feed Generation: Serve a personalized ranked list of articles for a user.
  3. Feedback Loop: Record user clicks or interactions to improve future recommendations.

Non-Functional Requirements

  1. Scale: Support 10 million daily active users and handle 10,000 new articles per hour.
  2. Latency: The personalized feed should load in under 200ms.
  3. Availability: The system must remain highly available, even if a few upstream news sources are down.

Scope of Response

Your design should address:

  • The high-level architecture diagram or flow of components.
  • Data models for articles, users, and interaction events.
  • How you would store and index the articles for fast retrieval.
  • The strategy for personalization (e.g., how to rank content).
  • Handling high traffic spikes and ensuring cache consistency.

Hints

Hint 1

Consider decoupling the ingestion pipeline from the user-facing feed generation service using a message queue.

Hint 2

Think about using a two-stage ranking approach: a coarse filtering layer to narrow down candidates and a refined ML-based model for top-k results.

Hint 3

Use a distributed search index (like Elasticsearch) for full-text queries and an in-memory cache (like Redis) for pre-computed user feeds.

Follow-up questions

  • How would you handle cold-start problems for new users who have no interaction history?
  • If the number of news sources increases tenfold, how would you scale the ingestion workers?
  • How would you design a feature to detect and deduplicate similar articles coming from different sources?