Katkat
Back

Reason About Duplicate Data and Scaling in Spark

CognitivSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

This problem tests the ability to design an idempotent, scalable data deduplication pipeline in Apache Spark while considering distributed systems constraints like data skew and state management.

Problem Statement

You are building a streaming pipeline for an advertising platform that ingests millions of events per second from various click trackers. Each event contains an event_id (UUID), timestamp, user_id, and payload.

Due to network retries at the source, the system receives duplicate events with the same event_id but potentially different arrival times. Your task is to design a Spark Structured Streaming process that filters duplicates and writes the cleaned data to an S3-based data lake.

Requirements

  1. Define a strategy to deduplicate events based on event_id while keeping the system performant at scale.
  2. Explain how you would handle 'late' events that arrive after the deduplication window has closed.
  3. Discuss how you would handle data skew if certain event_id prefixes appear significantly more frequently than others.
  4. Describe how to ensure the final sink is idempotent so that a Spark job retry does not result in duplicate records in the destination.

Constraints

  • The volume is 10M events/second.
  • The deduplication window required by the business is 24 hours.
  • The system must support backfilling historical data without duplicating existing production data.

Hints

Hint 1

Consider the memory trade-offs between using Spark's built-in `dropDuplicates` with watermarks versus managing an external state store like Redis or Cassandra.

Hint 2

For massive scale, think about how salting your keys (e.g., appending a random suffix to `event_id` before shuffling) can mitigate the performance bottlenecks caused by data skew.

Hint 3

To ensure idempotency in the sink, consider leveraging partition-based writing or transaction logs (like Delta Lake or Iceberg) to handle job restarts gracefully.

Follow-up questions

  • How would you monitor the pipeline to detect if the deduplication window is dropping valid data or consuming too much memory?
  • If the source system changes and starts sending events with new `event_id` formats, how would you evolve the schema and deduplication logic without stopping the pipeline?
  • Compare the performance implications of using a `StateStore` versus performing a join against a broadcast table for deduplication.