Katkat
Back

Design a Fault-Tolerant Video Generation Platform

OpenAISoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a distributed, asynchronous system capable of handling long-running, resource-intensive tasks while ensuring idempotency and fault tolerance in the event of partial component failure.

Problem Statement

Design a scalable, fault-tolerant video generation platform. Users submit prompts via a REST API, and the system performs computationally expensive video synthesis. A single request may involve multiple stages: prompt parsing, frame generation, interpolation, and encoding. Each stage can take minutes to hours and may be interrupted by node failures, preemption, or network partitions.

Requirements

  • Scalability: Handle varying demand while managing GPU/compute resource backpressure.
  • Fault Tolerance: Ensure that if a worker fails mid-generation, the system can resume or restart the specific stage without losing progress.
  • Idempotency: Prevent duplicate charges or inconsistent video artifacts if an operation is retried.
  • Observability: Track task status, resource utilization, and detect "zombie" jobs that are stuck without crashing.
  • Lifecycle: Support job cancellation by the user at any point in the pipeline.

Scale Assumptions

  • Peak traffic: 1,000 requests per minute.
  • Execution time: 5 to 30 minutes per video.
  • Persistence: Artifacts are stored in object storage (S3-like); metadata in a relational database.

Expectations

Provide a high-level architecture diagram (described in text), data schema for job tracking, a strategy for handling retries, and how the system prevents duplicate work (e.g., how to handle race conditions where two workers might claim the same job).

Hints

Hint 1

Use a durable message queue or task orchestrator to manage state transitions and decouple the submission API from the worker execution layer.

Hint 2

Implement a 'check-and-set' mechanism using atomic database operations or distributed locks to ensure only one worker processes a specific task at a given time.

Hint 3

Introduce a 'checkpointing' strategy where partial results are saved to persistent storage, allowing workers to resume stages rather than restarting the entire generation process.

Follow-up questions

  • How would you handle a situation where a worker node crashes right after completing a task but before updating the job status to 'complete'?
  • If the generation models become too large to fit in memory on a single GPU, how would you re-architect the task to support multi-node coordinated generation?
  • How would you implement a circuit breaker to stop accepting new jobs if the underlying GPU worker pool is consistently failing or exhausted?