Katkat
Back

Design trending queries ranking system

PerplexitySoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

This question tests the ability to design a high-throughput, real-time pipeline that identifies and ranks trending search queries, requiring trade-offs between freshness, accuracy, and system latency.

Problem Statement

Design a system that maintains a real-time list of the top 50 trending search queries globally for a search engine platform receiving 100,000 queries per second (QPS).

Functional Requirements

  • Provide a "Trending Now" list updated every 60 seconds.
  • The ranking must reflect sudden spikes in interest rather than just total historical volume.
  • The system must handle high burst traffic and maintain low read latency for the end-user API.

Non-Functional Requirements

  • Data must be accurate and reflect trends within a 1-minute window.
  • The system must be resilient to "query spam" (e.g., botnets trying to manipulate trends).
  • High availability and scalability as traffic grows.

Deliverables

  1. High-level architecture diagram (described in text).
  2. Data ingestion and processing pipeline design.
  3. Strategy for ranking "trending" queries vs. "popular" queries.
  4. Mechanisms to filter out noise and malicious query injections.

Hints

Hint 1

Consider using a streaming framework like Apache Flink or Spark Streaming to perform windowed aggregations on the incoming query stream.

Hint 2

Think about using a decaying counter or a statistical z-score approach to identify queries that are rising significantly faster than their baseline average.

Hint 3

To manage memory for millions of unique queries, consider using sketches like Count-Min Sketch or Top-K heavy hitters algorithms.

Follow-up questions

  • How would you adapt this system to support trending queries localized to specific geographic regions or user demographics?
  • If the system identifies a query that is a blatant attempt to spam the trending list, how do you handle it without manual intervention?
  • How does the choice of your storage layer affect the consistency and latency of the trending list API?