Katkat
Back

Design Direct File Uploads for a Chat Application

LegoraSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a scalable direct-to-object-store file upload pipeline for a chat platform, focusing on presigned authorization, durable lifecycle tracking, asynchronous validation, and orphan reconciliation.

Scenario

In a real-time team chat platform, users regularly share media and documents ranging from small avatars and screenshots to large video clips and archives. Routing multi-gigabyte daily file traffic through standard API servers introduces memory pressure, high bandwidth costs, and poor connection resilience.

Design an end-to-end direct-to-object-store upload architecture where clients upload directly to cloud storage (e.g., S3/GCS) while the backend retains strict access control, content verification, and lifecycle state management.

Requirements

Functional

  • Scoped Capabilities: Provide clients with secure, short-lived, size-bounded upload capabilities (e.g., presigned URLs) without exposing cloud storage master credentials.
  • Lifecycle State Machine: Durably track file state across transitions (INITIATED, UPLOADED, PROCESSING, ACTIVE, FAILED, EXPIRED).
  • Verification & Asynchronous Processing: Validate file integrity (checksums, MIME sniffing, malware scanning) and generate derivatives (thumbnails, transcoding) asynchronously before releasing the file into the chat.
  • Orphan Cleanup & Reconciliation: Detect and purge orphaned files created when clients initiate an upload but never finish or when backend processing fails.

Non-Functional

  • Security: Prevent arbitrary file type injection, overwrites of existing assets, and unauthorized access across chat channels.
  • Scalability: Support peak upload initiation rates with minimal backend resource utilization.
  • Reliability & Idempotency: Support upload retries and resume capabilities for large files without duplicating storage or database records.

Scale Assumptions

  • Daily Active Users (DAU): 20 million
  • Upload Volume: 5 million files per day (~200 uploads/second peak)
  • Payload Sizes: Images up to 25 MB, videos and archives up to 1 GB
  • Storage Retention: Files in active chat channels must persist indefinitely unless deleted by a user or retention policy.

Key Areas to Cover

  1. API & Handshake: The protocol flow between the client, chat backend, and object store for single-part and multipart uploads.
  2. Data Model: Schemas to track upload sessions, file metadata, channel associations, and lifecycle state.
  3. Event Pipeline: Architecture for handling storage events, background workers, and progress/completion notifications to the client.
  4. Failure Modes & Garbage Collection: Handling interrupted uploads, mismatched checksums, and delayed storage events.

Hints

Hint 1

Use presigned URLs with strict content-length and content-type conditions, and route uploads into an isolated 'quarantine' bucket path before validation.

Hint 2

Decouple file upload completion from message delivery by triggering asynchronous processing workers via object store event notifications.

Hint 3

Implement a scheduled reconciliation worker that compares database records in 'INITIATED' state past their TTL against the object store to delete abandoned multipart uploads and clean up metadata.

Follow-up questions

  • How would you handle multipart uploads for 1 GB video files while allowing clients to resume interrupted uploads without re-uploading completed parts?
  • How do you mitigate rate-limit abuse where a malicious user requests millions of presigned upload URLs without uploading files?
  • If an object store notification message is dropped or duplicated by the message queue, how does your system guarantee exactly-once state transition to 'ACTIVE'?