Katkat
Back

Design TikTok Data Engineering Systems

AppleData Engineer · Technical Screen
MediumData EngineerTechnical ScreenSystem Design

Last updated

This problem tests the ability to design an end-to-end data pipeline for high-throughput mobile app event logging, focusing on ingestion, transformation, and storage optimization.

Scenario

You are designing a data engineering pipeline for TikTok's core engagement metrics. The system needs to capture events such as 'video_view', 'like', and 'share' from millions of global users.

Requirements

  • Ingestion: Handle 100,000 events per second.
  • Storage: Support both sub-second latency lookups for trending metrics and analytical SQL queries over multi-year historical data.
  • Reliability: Ensure data consistency and handle transient network failures or traffic spikes without losing events.

Task

Design the architecture for this pipeline. Your response should address:

  1. Data Ingestion: How do you buffer and ingest these events?
  2. Processing: How do you handle stream processing (e.g., event deduplication, windowing for trending hashtags)?
  3. Data Modeling & Storage: Propose a storage strategy for the analytics warehouse (e.g., partitioning, indexing, columnar storage formats).
  4. Trade-offs: Discuss how you balance the latency of dashboards with the cost of maintaining long-term historical storage.

Hints

Hint 1

Consider using a distributed message queue like Kafka to act as a buffer between mobile clients and downstream processing.

Hint 2

For the storage layer, compare the pros and cons of using a data lakehouse architecture with Parquet/Iceberg vs. a specialized OLAP database for real-time aggregation.

Hint 3

Think about how to implement partitioning in your warehouse—is it better to partition by user_id or by event_timestamp given the query patterns?

Follow-up questions

  • How would you handle late-arriving events that are processed after their event window has already closed?
  • If the 'like' count dashboard is showing inaccurate numbers due to data skew in high-profile viral videos, how would you refactor the aggregation logic?
  • How would you ensure GDPR compliance regarding user data deletion requests within your storage architecture?