Katkat
Back

Design a Text-to-Video Generation Platform (Sora-style)

OpenAISoftware Engineer · Onsite
HardSoftware EngineerOnsiteML System Design

Last updated

This problem evaluates your ability to design a distributed ML infrastructure that handles high-latency, resource-intensive inference jobs while ensuring scalability, reliability, and content safety.

Problem: Designing a Text-to-Video Generation Platform

Design a scalable system that allows users to submit text prompts and receive high-quality, 10-second video clips. Generating a single clip takes approximately 2-5 minutes of GPU compute time on top-tier hardware (e.g., H100s).

Functional Requirements

  • Users submit prompts via an API and receive a job ID.
  • Users can poll for status or receive a webhook notification upon completion.
  • The system must maintain a history of generated videos for user retrieval.

Non-Functional Requirements

  • Scale: Handle 10,000 requests per hour.
  • Reliability: Ensure no jobs are lost due to node failure or transient errors.
  • Safety: Implement a content moderation layer to filter out harmful prompts and generated content before delivery.
  • Latency: While generation is slow, the system must provide immediate feedback on job acceptance and queue position.

Your Design Should Cover:

  1. High-Level Architecture: API gateway, job orchestration, and communication between services.
  2. Job Queue & Scheduling: How to manage a queue of long-running tasks and ensure fair scheduling across a limited GPU fleet.
  3. Data Pipeline: Managing large video files (storage, CDN, and metadata).
  4. Failure Handling: How to handle worker crashes during the middle of a generation process.

Constraints & Assumptions

  • Storage: Videos average 50MB each.
  • Compute: Model weights are multi-gigabyte, requiring careful loading/caching strategies on GPU nodes.
  • You have an existing pre-trained model as a 'black box' function generate_video(prompt).

Hints

Hint 1

Consider using a durable message queue (like RabbitMQ or Kafka) to act as a buffer between the API tier and the GPU worker pool.

Hint 2

Think about the difference between 'request-response' patterns and 'asynchronous job' patterns, and why the latter is essential here.

Hint 3

Address how to implement 'autoscaling'—not just based on CPU, but based on GPU utilization and queue depth.

Follow-up questions

  • How would you optimize the system if 80% of your users submit similar prompts that could potentially be cached?
  • How would you implement a 'priority' system where Pro users get their videos generated before free-tier users?
  • If the content moderation layer flags a video, what is the most efficient way to stop the compute process to save costs?