Katkat
Back

Design a news aggregation and feed service

RipplingSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed system capable of ingesting high-frequency news feeds from multiple sources and serving personalized, low-latency content to millions of users.

Problem Statement

Design a news aggregation and feed service that collects news articles from various RSS feeds and APIs, stores them, and provides a personalized "For You" feed to millions of daily active users.

Functional Requirements

  1. Ingest articles from thousands of external sources periodically.
  2. Allow users to follow topics (e.g., "Tech", "Finance") and specific publishers.
  3. Serve a personalized feed of articles based on user preferences and recency.
  4. Ensure the feed is updated with new content within a reasonable delay.

Non-Functional Requirements

  1. High availability and low-latency feed retrieval (e.g., < 200ms).
  2. Scalability: Support 10 million daily active users and 100k+ new articles daily.
  3. Reliability: Ensure no articles are lost during ingestion and the system can handle traffic spikes.

Requirements for your design:

  • High-level architecture diagram (describe components like ingestion workers, databases, and caches).
  • Data model schema (SQL or NoSQL choices).
  • Strategies for feed generation (Push model vs. Pull model vs. Hybrid).
  • Handling of spikes in content volume or user traffic.
  • Trade-offs regarding data consistency (is it okay if a user sees an article 10 seconds late?).

Hints

Hint 1

Consider decoupling the ingestion layer from the serving layer using a message queue to handle throughput variations.

Hint 2

For the feed generation strategy, analyze the trade-offs between pre-computing feeds for active users versus computing them on-the-fly.

Hint 3

Think about using a caching strategy (e.g., Redis) to store pre-generated feed IDs and a separate store for actual article metadata to minimize latency.

Follow-up questions

  • How would you implement a deduplication mechanism to ensure that the same article from multiple sources isn't shown multiple times?
  • If the service suddenly experiences a 10x spike in traffic during a major global event, which components would likely fail first and how would you scale them?
  • How would you integrate a simple ranking algorithm to sort articles in the feed beyond just 'latest'?