Katkat
Back

Design a personalized news aggregator

RipplingSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question evaluates a candidate's competence in large-scale system design, covering concepts such as high-throughput API ingestion, idempotent processing and deduplication, indexing and storage, caching, and low-latency personalized feed generation.

Design a Google News-style personalized news aggregator.

The system continuously collects articles from many third-party publishers by polling their APIs (assume most publishers expose a REST/JSON feed and do not offer webhooks or streaming). Users can:

  • Browse a feed of current news.
  • Follow specific publishers.
  • Select preferred content categories (e.g. Business, Sports, Technology, Entertainment).

Personalization is deliberately limited to two explicit signals only: the publishers a user follows and the categories they select. There is no implicit/behavioral personalization (no click-history models, no embeddings of the user). Design the end-to-end system, covering ingestion, storage, feed generation, and operations.

Constraints & Assumptions

State your assumptions explicitly; reasonable numbers are fine. A typical framing:

  • Publishers: ~$10{,}000$ sources, each polled on its own schedule; per-publisher rate limits and SLAs vary.
  • Article volume: on the order of $1$–$5$ million new/updated articles per day (tens of articles/sec average, with spikes around major events).
  • Users: ~$100$M registered, ~$10$M daily active; read-heavy ($\gg 100{:}1$ read:write at the feed layer).
  • Feed latency target: $p99 < 200$ ms for a cached first page; freshness target: a newly published article should be eligible for feeds within a few minutes (bounded by publisher polling cadence and rate limits).
  • Availability: feed serving should stay up and degrade gracefully if some publishers or the ranking layer fail; ingestion may lag without breaking reads.
  • Personalization signals are explicit only (followed publishers + selected categories) — no behavioral/ML personalization in scope.

Clarifying Questions to Ask

  • How many publishers, and do any support webhooks/push or conditional requests (ETag, cursors), or is it pure polling?
  • What freshness SLA matters — must breaking news surface in seconds, or are minutes acceptable?
  • Read/write ratio and DAU scale — does the feed need precomputation, or is compute-on-read sufficient?
  • Is full-text keyword search of articles in scope, or only category/publisher filtering?
  • How aggressive should deduplication be — exact duplicates only, or near-duplicate clustering of the same story across publishers?
  • Are personalization signals truly limited to follows + categories, or should the design leave room for future engagement-based ranking?

Hints

Hint 1

Where to start: This system has two activities with very different shapes: continuously pulling articles from thousands of third-party APIs, and answering a feed request fast. Ask yourself which one must be low-latency and which can run in the background — and what that implies about how you'd separate them and what sits between them.

Hint 2

What does "personalized" actually require here?: The only personalization signals are *followed publishers* and *selected categories* — no ML, no per-user models. Given that, ask: how much of the work to answer a feed request is genuinely specific to *one* user, versus the same for everyone who follows a given publisher or picks a given category? Let that distinction drive where you spend compute and what you cache.

Hint 3

Where will the interviewer push?: Two parts of this problem are deceptively hard. First, *when* and *how* do you poll thousands of APIs without either wasting requests or missing fresh stories — what state would you keep per publisher to do better than a fixed timer? Second, the same story is filed by many outlets — what does "the same article" even mean, and is one notion of sameness enough? Have a concrete plan for each before going deep on the rest.

Hint 4

Polling re-fetches the same article: Polling is inherently at-least-once: overlapping polls and retries will hand you the same article more than once. Before you store anything, ask what property each ingestion stage needs so that seeing an article twice is harmless — and what identity you'd key that on.

Follow-up questions

  • A breaking story is published by 50 outlets within 2 minutes. Walk through exactly how your dedup pipeline clusters them and picks the one article a user sees — and what happens to the score of the cluster.
  • A single popular publisher's API starts returning `429`/`500`. How does your system protect freshness for *everyone else* and recover that publisher without manual intervention?
  • The product team now wants to add engagement-based ranking (clicks, dwell time) on top of follows/categories. What in your design has to change, and what stays the same?
  • You're told the $p99$ feed latency is fine but the *freshness* p99 is 8 minutes and the SLA is 2 minutes. Where do you look first, and which knobs do you turn?