Katkat
Back

Design Trending Collaborative Files

FigmaSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This problem tests the ability to design a distributed streaming architecture that aggregates real-time user activity while maintaining strict privacy and performance requirements.

Problem Statement

Design a system to identify and rank the "Top 10 Trending Files" across a collaborative design platform. The platform generates billions of events daily (e.g., clicks, edits, comments).

Functional Requirements

  • Rank files based on a 'trending' score calculated over a sliding 60-minute window.
  • Ensure the ranking is updated at least every minute.
  • Prevent private or restricted files (those not shared with a specific user) from appearing in their personalized trending feed.

Non-Functional Requirements

  • Handle high write throughput (millions of events per second).
  • Minimize latency between event occurrence and dashboard visibility.
  • High availability and eventual consistency are acceptable for the ranking list.

The Task

Provide a high-level architecture that addresses:

  1. Data Ingestion: How to collect and buffer raw events.
  2. Aggregation: The processing pipeline for windowed counts.
  3. Storage: Where to store the top-k results and how to serve them efficiently.
  4. Privacy: How to ensure users only see trending files they have permission to access.
  5. Abuse Control: Mechanisms to prevent a single user or bot from artificially inflating a file's trend score.

Hints

Hint 1

Consider using a stream processing framework like Apache Flink or Kafka Streams to handle sliding windows and state management.

Hint 2

For the privacy requirement, think about decoupling the global trending list from the user-specific view by using a filter-on-read approach or a Bloom filter-based check.

Hint 3

To handle abuse, consider incorporating a weighted scoring mechanism that de-prioritizes events from new accounts or repeated actions from the same IP address.

Follow-up questions

  • How would you handle late-arriving events that occur due to network partition or mobile client offline states?
  • If the number of collaborative files grows to billions, how would you optimize the storage and lookup for the top-k ranking service?
  • How would you measure the effectiveness of your 'trending' algorithm and detect if it is being gamed?