Katkat
Back

Design a Large-File Upload and Analysis Flow

PinterestSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Tests the ability to design an end-to-end media ingestion system supporting chunked resumable uploads, asynchronous analysis pipelines, idempotency, and real-time client notifications.

Scenario

Content creators on a media-rich platform (such as Pinterest) frequently upload large video and high-resolution image files (ranging from 100MB up to 5GB) to publish as rich Pins. Once uploaded, these files must undergo an asynchronous pipeline consisting of media validation, virus scanning, multi-resolution transcoding, copyright detection, and ML-based content tagging. You need to design the architecture that handles file ingestion, coordinates downstream processing, and reliably informs the creator once the file is ready to publish.

Functional Requirements

  • Resumable Uploads: Support chunked, resumable uploads to withstand unstable mobile or web network connections.
  • Direct-to-Storage Ingestion: Upload media payload directly to object storage without choking application servers.
  • Idempotency: Prevent duplicate processing if clients retry chunks or re-submit upload completion requests.
  • Asynchronous Analysis Pipeline: Coordinate independent processing steps (transcoding, safety checks, tagging) with intermediate state tracking.
  • Reliable Notification: Notify the creator client in near-real-time once processing finishes or if an unrecoverable failure occurs.

Non-Functional Requirements & Scale

  • Traffic Volume: 20 million upload attempts per day (~250 uploads/sec average, 1,000 uploads/sec peak).
  • File Sizes: Average 200MB, maximum 5GB.
  • Durability & Availability: 99.99% service availability; 11 9s durability for stored assets.
  • Analysis Latency: End-to-end processing should complete within 3–5 minutes for a 500MB video under normal load.

What Your Design Should Cover

  1. Upload Protocol & API: Detailed flow for initiating sessions, uploading chunks via pre-signed URLs, and finalizing uploads.
  2. Storage & Metadata Schema: Data structures to track chunk progress, file status, and analysis task states.
  3. Asynchronous Processing Pipeline: Decoupled worker architecture, message queues, and dependency management across pipeline stages.
  4. Failure Modes & Idempotency: Strategies for handling aborted uploads, duplicate callbacks, worker crashes, and partial pipeline failures.
  5. Notification System: Mechanism used to push state transitions to active clients (e.g., WebSockets, SSE, or polling fallback).

Hints

Hint 1

Use pre-signed S3 or GCS multipart upload URLs generated by an API gateway to keep large binary traffic out of your core application servers.

Hint 2

Track upload sessions and chunk checksums (e.g., MD5 or SHA-256) in a relational or document store to guarantee idempotent retries.

Hint 3

Structure downstream analysis using an event-driven orchestrator or distributed task DAG where individual workers update a centralized state machine.

Follow-up questions

  • How would you clean up orphaned chunks and incomplete uploads from abandoned user sessions without incurring runaway storage costs?
  • How can you leverage global edge infrastructure (such as CDN edge points of presence or AWS S3 Transfer Acceleration) to reduce upload latency for international users?
  • If an expensive downstream task like video transcoding succeeds but a subsequent ML safety check times out, how do you avoid re-running transcoding during retries?