Katkat
Back

Design a Streaming Top-Ten Sales Dashboard

eBaySoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed, low-latency streaming pipeline that aggregates high-velocity sales data into a real-time leaderboard.

Problem Statement

Design a real-time streaming system that calculates the "Top 10" most sold items and categories across a global marketplace. The system must process millions of events per second with high availability and low latency.

Requirements

  • Functional Requirements:
    • Maintain a real-time leaderboard of the top 10 items (by units sold) and top 10 categories.
    • Support a dashboard that refreshes the leaderboard every few seconds.
  • Non-Functional Requirements:
    • Scale: Handle 1 million events per second peak load.
    • Accuracy: Handle out-of-order events and potential duplicates without significantly impacting latency.
    • Availability: The system must recover from node failures without losing significant state.

Expectations

Provide a high-level architecture diagram or description covering:

  1. Data Ingestion: How to receive events from distributed transaction services.
  2. Aggregation Strategy: Explain the trade-off between centralized aggregation vs. multi-stage (partitioned) aggregation.
  3. Storage/State Management: How to maintain running counts while ensuring the system can recover after a crash.
  4. Downstream Consumption: How the frontend dashboard queries the aggregated data (push vs. pull models).

Hints

Hint 1

Consider using a distributed streaming platform like Apache Kafka for ingestion and a stream processing engine like Apache Flink for windowed aggregation.

Hint 2

Think about using a two-tier aggregation approach: local aggregation at the partition level followed by a global merge to reduce network contention.

Hint 3

Address duplicate data by using idempotent writes or deduplication logic at the ingress point based on a unique transaction ID.

Follow-up questions

  • How would you handle a sudden surge in traffic (e.g., Black Friday) to ensure the leaderboard doesn't lag?
  • If we need to support a 'Top 10' for the last 60 minutes rolling window, how does your aggregation strategy change?
  • How would you implement 'exactly-once' processing semantics in your pipeline?