Katkat
Back

Design a 200-Terabyte Media Migration and Training Input Pipeline

AppleSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenML System Design

Last updated

This problem tests your ability to design a high-throughput data ingestion pipeline, focusing on network bottlenecks, storage strategies, and synchronization between on-premises assets and cloud-based training clusters.

Scenario

You are tasked with migrating 200 TB of raw image and video data from a legacy on-premises data center to a cloud-based GPU training cluster. The goal is to prepare this data for a distributed deep learning training job. The transfer must be completed within 72 hours. Once the data arrives, it needs to be pre-processed (e.g., resizing, normalization, augmentation) and made available to the training nodes with minimal I/O latency.

Requirements

  1. Migration Strategy: Explain how you would physically or logically move 200 TB of data to the cloud within the time constraint while accounting for existing bandwidth limitations.
  2. Pipeline Architecture: Describe the pipeline from the cloud storage bucket to the GPU training cluster, including data formats (e.g., TFRecord, Petastorm, or raw files) and any caching mechanisms.
  3. Data Pre-processing: How do you handle the compute-intensive pre-processing step without starving the GPUs? Consider using a distributed pre-processing framework.
  4. Failure Handling: How do you ensure data integrity and track progress across the 200 TB transfer?

Constraints

  • The on-premises network has a 1 Gbps uplink dedicated to this task.
  • Training jobs require high-speed local disk access (NVMe) to avoid GPU starvation.
  • Training must be reproducible, meaning dataset versions must be tracked.

Hints

Hint 1

Consider physical data transfer appliances (like AWS Snowball or Google Transfer Appliance) versus network-based transfer given the 1 Gbps constraint.

Hint 2

Evaluate the trade-off between keeping data in object storage versus mounting a high-performance distributed file system like FSx for Lustre.

Hint 3

Discuss how to decouple the ingestion pipeline from the training pipeline using a staging area that handles format conversion and sharding.

Follow-up questions

  • How would you handle the training process if the 200 TB dataset is too large to fit entirely onto the local NVMe storage of the GPU nodes?
  • If the source data is updated frequently, how would you change your design to support continuous synchronization instead of a one-time migration?
  • How would you implement monitoring to detect bottlenecks if the GPU utilization drops below a certain threshold during training?