Katkat
Back

Design News Pull Scheduling, Article Storage, and Topic Reads

RipplingSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a scalable topic-based news aggregation system covering distributed feed scheduling, idempotent article ingestion, stable pagination, and a low-latency read path.

Scenario

Design a news aggregation platform that continuously pulls content from external sources (RSS feeds, news APIs), categorizes articles by topic, stores them reliably, and serves topic-filtered news feeds to end users.

Requirements

Functional Requirements

  • Feed Scheduling & Ingestion: Schedule periodic pulls from tens of thousands of external sources at varying frequencies (e.g., 5 to 60 minutes) while honoring source rate limits.
  • Idempotent Ingestion: Store articles durably and prevent duplicate entries even when pull jobs retry or external sources re-publish existing articles.
  • Topic Association: Tag and index articles under one or more topics (e.g., payroll, benefits, compliance).
  • Topic Feed Reads: Allow users to query articles by topic, ordered chronologically, using stable pagination (no duplicated or skipped items as new articles arrive).

Non-Functional Requirements

  • Read Latency: p99 latency under 50ms for topic reads.
  • Availability & Durability: Zero data loss for ingested articles; high availability for read queries.
  • Fault Tolerance: Isolation of slow or failing third-party sources so they do not exhaust worker capacity.

Scale Assumptions

  • Sources: 50,000 registered feeds.
  • Write Volume: ~5 million articles ingested per day (average ~60 articles/sec, peak ~300 articles/sec).
  • Retention: 100 million total articles retained over 12 months.
  • Read Traffic: 10,000 QPS on topic feed endpoints.
  • Topics: ~500 predefined topics, with skewed interest (top 20 topics account for 80% of read traffic).

What Your Design Should Cover

  1. API Design: Endpoints for querying topic feeds with pagination.
  2. Scheduler Architecture: Mechanism to schedule, dispatch, and rate-limit feed pulls at scale across distributed workers.
  3. Storage & Deduplication: Schema, indexing strategy, and deduplication keys (e.g., URL canonicalization, content hashes) to guarantee idempotent writes.
  4. Read Path & Caching: Architecture to handle 10,000 QPS with sub-50ms latency, including cache structure (e.g., Redis Sorted Sets) and keyset/cursor-based pagination design.
  5. Trade-offs: Latency vs. data freshness, write amplification from fan-out indexing, and offset vs. cursor pagination.

Hints

Hint 1

Use deterministic unique constraints (such as a normalized URL hash or content signature) to make article ingestion idempotent under network retries.

Hint 2

Avoid SQL `OFFSET` pagination because concurrent writes cause item drift; instead, design a cursor based on `(published_at, article_id)`.

Hint 3

For the read path, pre-materialize topic timelines in an in-memory store like Redis sorted sets (ZSET) scored by timestamp to meet strict read latency targets.

Follow-up questions

  • How would you handle third-party feeds that temporarily fail, return HTTP 429, or respond very slowly without consuming all scheduler worker threads?
  • If an article is edited or retracted hours after being ingested, how does your system handle cache invalidation and update propagation?
  • How would your scheduling architecture change if the number of feeds scaled 100x from 50,000 to 5,000,000?