Katkat
Back

Design a News Aggregation Service

RipplingSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a scalable news aggregation platform that ingests high-volume publisher feeds, clusters and deduplicates coverage into cohesive stories, and serves fresh, personalized feeds with sub-second latency.

Scenario

Design a global news aggregation platform (similar to Google News or Apple News) that continuously fetches articles from thousands of publisher RSS feeds and REST APIs, deduplicates identical or syndicated stories, clusters coverage around shared events, and serves personalized, ranked news streams to end users.

Requirements

Functional Requirements

  • Feed Ingestion & Normalization: Ingest articles across diverse publisher sources (RSS/Atom, webhooks, scrapers). Normalize disparate schemas and track article revisions/updates over time.
  • Deduplication & Clustering: Detect exact and near-duplicate articles (syndicated content) and cluster related articles into a single multi-perspective "story topic".
  • Serving & Ranking: Serve a paginated, personalized home feed to users with fresh stories, balanced diversity across publishers and topics, and low latency.
  • Content Deletions & Takedowns: Propagate publisher retraction requests or copyright takedowns downstream to search indices and edge caches within minutes.

Non-Functional Requirements

  • High Availability: Read path 99.99% availability; ingestion path resilient to external source downtime.
  • Low Latency: P95 read response times under 100ms.
  • Politeness & Rate Limiting: Respect publisher robots.txt, crawl delays, and rate limits.
  • Auditability & Replay: Ability to reprocess raw article archives to test new clustering/ranking algorithms or rebuild corrupted indices.

Scale Assumptions

  • Publishers: 50,000 distinct registered sources.
  • Article Volume: 10 million new or updated articles ingested per day (~120 articles/sec average, 1,000 articles/sec peak).
  • Read Traffic: 50 million Daily Active Users (DAU), 200 million feed read requests per day (~2,500 requests/sec average, 10,000 requests/sec peak).
  • Retention: Raw articles and revisions retained for 5 years; real-time serving index focuses on active stories from the last 14 days.

Key Areas to Cover

  1. High-Level Architecture: Ingestion pipeline, processing workers, storage tiers, and serving layer.
  2. Data Model & Schema: Storage strategy for raw documents, content revisions, clusters, and user interactions.
  3. Clustering & Deduplication Pipeline: Algorithms and architecture for near-duplicate detection (e.g., MinHash/SimHash) and incremental clustering (embeddings, vector search).
  4. Feed Generation & Caching: Cache hierarchy (CDN, Redis, edge), personalized ranking trade-offs (fan-out vs. pull-on-read), and cache invalidation strategies.

Hints

Hint 1

Decouple the ingestion crawlers from downstream text processing using an append-only event stream (e.g., Kafka) to absorb publisher burstiness and allow pipeline replay.

Hint 2

For near-duplicate detection and story clustering, consider two-tiered approaches: fast fingerprinting (SimHash/MinHash) for syndication and vector embeddings for semantic grouping.

Hint 3

To handle rapid publisher edits and takedowns, maintain an explicit state machine for article versions and use tombstone events to invalidate edge and application caches.

Follow-up questions

  • How does your clustering architecture handle breaking news events where thousands of articles are published on the same topic within a few minutes?
  • How would you enforce diversity in the final feed to prevent a single dominant publisher or topic from saturating a user's top-20 view?
  • If a ranking bug corrupts cluster assignments for the past 24 hours, what is your strategy for deterministic re-clustering and zero-downtime cache restoration?