Katkat
Back

Design a Sliding-Hour Top-100 User Event Pipeline

OracleSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a real-time event-time aggregation pipeline to compute the top 100 most active users over a trailing 60-minute sliding window at scale.

Scenario

An enterprise cloud platform requires an operational analytics dashboard to identify the top 100 most active users across the platform over a trailing 1-hour window. The downstream abuse prevention and capacity monitoring services query these rankings in real time.

Design a resilient, high-throughput stream processing pipeline that ingests raw user actions and continuously outputs the global Top-100 users by event count for the trailing 60 minutes.

Requirements

Functional Requirements

  • Ingest structured events containing (user_id, event_type, timestamp, event_id).
  • Continuously maintain and serve the global Top-100 users with the highest event counts over the trailing 60 minutes, evaluated by event time.
  • Expose an API endpoint GET /v1/top-users?window=60m to retrieve the current ranking.

Non-Functional Requirements

  • Freshness: The top-100 list should update at least once every 10 to 60 seconds (slide step).
  • Correctness: Support out-of-order events within a bounded delay (e.g., up to 5 minutes late).
  • High Availability & Fault Tolerance: Zero data loss for state; recovery from worker failures must not reset window counts.
  • Scalability: Handle sudden traffic bursts and uneven event distributions across users.

Scale Assumptions

  • Throughput: 100,000 events/second normal, peaking at 300,000 events/second.
  • Cardinality: Approximately 10 million distinct active users in any given 1-hour window.
  • Average event size: 500 bytes.

What Your Design Should Cover

  1. Data Ingestion & Stream Processing Engine: Ingestion layer and choice of streaming framework (e.g., Apache Kafka + Apache Flink).
  2. Window Management & State Representation: How to maintain trailing 1-hour state efficiently without re-aggregating raw events every slide (e.g., tumbling sub-buckets vs. keyed state).
  3. Aggregation Hierarchy: Strategies for preventing hot-key bottlenecks and aggregating partial local top-K results to a single global top-K.
  4. Event Time, Watermarks & Deduplication: Handling late arrivals, network re-transmissions, and exactly-once processing semantics.
  5. State Backend & Fault Tolerance: Checkpointing, state size estimation, and recovery strategies.

Hints

Hint 1

Decompose the continuous 60-minute sliding window into discrete sub-buckets (such as 1-minute tumbling windows) to avoid storing every raw event in keyed state.

Hint 2

Avoid routing all user counts to a single aggregator node; use a two-phase aggregation with local top-K candidate extraction before global merging.

Hint 3

Set an event-time watermark threshold to handle late-arriving events while defining a cut-off point to safely evict expired bucket state.

Follow-up questions

  • How does your design adapt if a botnet causes a single user key to account for 30% of total incoming traffic?
  • If the business demands exact deduplication over a 24-hour deduplication window before aggregation, how does that impact storage and state backend sizing?
  • How would you design an offline batch reconciliation pipeline to verify and correct state drift in the streaming pipeline?