Katkat
Back

Design a Text-to-Video Generation Service

OpenAISoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenML System Design

Last updated

This question tests the ability to design a scalable, reliable asynchronous inference pipeline for resource-intensive generative models, focusing on task orchestration, GPU scheduling, and latency management.

System Design: Text-to-Video Generation Service

Design a service that allows users to submit text prompts and receive high-quality generated video clips.

Requirements

  • Functional: Users submit a prompt and a duration preference (e.g., 5s, 10s). The system returns a unique Job ID, polls for status, and provides a download URL upon completion.
  • Non-Functional: The system must handle high burst traffic. Inference is slow (seconds to minutes per video). Costs are dominated by high-end GPU utilization. High availability and fault tolerance are required.

Considerations

  1. API Design: Define the asynchronous lifecycle (Submission, Status Polling/Webhooks, Retrieval).
  2. Orchestration: How do you manage a queue of long-running inference jobs across a cluster of heterogenous GPU instances?
  3. Scaling: How do you handle a sudden 10x spike in requests?
  4. Storage: How is the raw video data stored and served?
  5. Failure Handling: How do you detect and recover from worker crashes during inference?

Deliverables

Provide a high-level architecture diagram (described in text), an explanation of the data flow, and a discussion of the trade-offs between throughput and latency.

Hints

Hint 1

Consider decoupling the API gateway from the GPU workers using a message broker like Kafka or RabbitMQ to buffer requests.

Hint 2

Use a durable state store (like Redis or DynamoDB) to track job metadata and transition states between 'queued', 'processing', and 'completed'.

Hint 3

Think about how to implement a priority queue or a job scheduler that matches specific model requirements to the available hardware capabilities.

Follow-up questions

  • How would you implement an auto-scaling policy for your GPU worker pool based on queue depth?
  • If inference is too expensive for all users, how would you design a rate-limiting or tiered-priority system?
  • How would you cache previous generations to save GPU costs for identical or highly similar prompt inputs?