Katkat
Back

Design Near-Real-Time Top Songs by Country

ByteDanceSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests your ability to design a distributed streaming data pipeline that balances high-throughput ingestion with low-latency analytical queries for country-specific rankings.

Problem Statement

Design a system that tracks and provides the top 10 songs played per country, updated every 10 minutes. The system receives high-volume song listen events (e.g., user_id, song_id, country_code, timestamp).

Requirements

  • Functional:
    • Ingest high-volume stream of listen events.
    • Maintain a "Top 10" leaderboard for every country, recalculated every 10 minutes.
    • Provide a read API that returns the current Top 10 for a given country.
    • Support "replay" functionality: if a calculation fails or needs adjustment, you must be able to re-process historical data.
  • Non-Functional:
    • Idempotency: Ensure that processing the same event multiple times does not skew the rankings.
    • Consistency: Eventual consistency is acceptable given the 10-minute refresh window.
    • Scalability: System must handle millions of concurrent listeners across 200+ countries.

Expectations

Your design should cover:

  1. Data Pipeline: Describe the end-to-end flow from event ingestion to the final leaderboard store.
  2. Storage Strategy: Choose appropriate databases for raw logs and aggregated rankings.
  3. Scaling: Explain how you handle spikes in traffic and partition data to avoid hotspots.
  4. Idempotency/Replay: Describe how you guarantee uniqueness and manage the re-processing of old windows.

Hints

Hint 1

Consider using a distributed message queue like Kafka to act as a buffer for raw event streams.

Hint 2

Think about using a stream processing framework (e.g., Apache Flink) to perform windowed aggregations and state management for each country.

Hint 3

Use a fast Key-Value store or a cache like Redis to serve the final Top 10 rankings for low-latency lookups.

Follow-up questions

  • How would you change your design if the refresh interval was reduced from 10 minutes to 10 seconds?
  • How do you handle 'viral' songs where millions of listens arrive for a single song ID in a single window, potentially causing a performance bottleneck?
  • What strategies would you implement to ensure data integrity if the message queue experiences a partial outage?