Katkat
Back

Design a Fault-Tolerant Distributed Job Scheduler

RobinhoodSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed, fault-tolerant system capable of scheduling and executing tasks reliably at scale, emphasizing durability, state management, and consistency.

System Design: Distributed Job Scheduler

Design a distributed job scheduler that supports one-time and recurring background tasks. The system must ensure that jobs are executed reliably, even if the scheduler nodes or worker nodes crash.

Functional Requirements

  • Submission: Clients can submit tasks with an execution timestamp and a payload.
  • Scheduling: Tasks must be executed at or after their scheduled time.
  • Reliability: No task should be lost; if a worker fails while processing, the task must be retried.
  • Deduplication: Ensure the same job ID is not executed multiple times if submitted concurrently.
  • Cancellation: Allow users to cancel pending tasks.

Non-Functional Requirements

  • Scale: Handle 10,000 job submissions per second and 1 million pending tasks.
  • Durability: Task state must persist across service restarts.
  • Fault Tolerance: System should handle node failures without manual intervention.

Deliverables

  1. API Definitions: Define endpoints for submitting, cancelling, and querying job status.
  2. Data Model: Describe the database schema or storage strategy for tasks.
  3. Architecture: Provide a high-level diagram or description covering the scheduler, task queue, worker communication, and storage layer.
  4. Failure Handling: Explain how the system handles worker crashes during job execution (e.g., leases, heartbeats) and scheduler crashes during task dispatch.

Hints

Hint 1

Consider using a distributed database or a transactional outbox pattern to ensure atomicity between job storage and dispatch.

Hint 2

Think about using a 'lease' mechanism where workers acquire a task with a time-to-live (TTL) to prevent orphaned tasks in case of worker failure.

Hint 3

For scaling, consider sharding tasks by their execution time or utilizing a partitioned message queue to prevent bottlenecking.

Follow-up questions

  • How would you handle backpressure if the number of incoming jobs significantly exceeds your worker pool capacity?
  • How do you ensure idempotency for external side effects triggered by these jobs (e.g., calling a third-party payment API)?
  • What strategies would you use to support high-precision scheduling (e.g., sub-second delay requirements) versus high-throughput batch processing?