Katkat
Back

Build a One-Pass Data Cleaning Pipeline

xAIData Engineer · Technical Screen
MediumData EngineerTechnical ScreenSystem Design

Last updated

This problem tests your ability to design a memory-efficient, scalable data streaming pipeline that processes raw web-scraped documents into a high-quality dataset for model pretraining.

System Design: One-Pass Data Cleaning Pipeline

You are tasked with building a streaming pipeline to ingest 100TB of raw HTML-based web documents and output a clean, deduplicated, and filtered corpus suitable for training a Large Language Model. The system must process data in a single pass to minimize latency and cloud storage costs.

Functional Requirements

  1. Extraction: Convert raw HTML to plain text while stripping boilerplate (navigation, ads, scripts).
  2. Quality Filtering: Discard documents that contain excessive non-alphanumeric characters, have low word counts, or fail a language identification check (only keep English).
  3. Deduplication: Perform fuzzy deduplication to remove near-duplicate documents within the stream.
  4. Telemetry: Emit real-time metrics on throughput, filter rejection rates, and document length distributions.

Non-Functional Requirements

  • Scalability: The system should be horizontally scalable to handle 100TB+.
  • Memory Efficiency: Avoid loading the entire dataset into memory; assume worker nodes have 32GB RAM.
  • Fault Tolerance: If a node fails, the pipeline should resume without reprocessing the entire dataset.

Your Task

Design the architecture for this pipeline. Specify the technologies used for ingestion, processing, and storage. Address how you will perform fuzzy deduplication without a global full-table lookup, and explain how you will track processing metrics in real-time.

Hints

Hint 1

Consider using Bloom filters or MinHash/LSH (Locality Sensitive Hashing) to approximate fuzzy deduplication within a streaming context.

Hint 2

Decouple the extraction and filtering logic from the ingestion layer by using a message queue or distributed streaming platform like Apache Kafka or AWS Kinesis.

Hint 3

Utilize a time-series database like Prometheus or InfluxDB to capture the telemetry data being emitted by your worker nodes.

Follow-up questions

  • How would you modify this pipeline if you needed to perform 'pii-masking' (Personally Identifiable Information) before the data hits the storage layer?
  • If the deduplication needs to be global (not just per-batch), how does your architecture change regarding state management?
  • What strategies would you implement to detect and handle data drift or quality degradation if the source web-scraper configuration changes over time?