Katkat
Back

Design a Reliable Job Scheduler

NuroSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a fault-tolerant distributed job scheduler capable of handling delayed execution, job cancellation, timeout detection, and automatic worker retries.

Scenario

Autonomous vehicle fleet platforms rely heavily on background asynchronous tasks for telemetry processing, remote vehicle diagnostics, simulation batch runs, and over-the-air firmware verification. You are tasked with designing a robust, distributed job scheduler that reliably triggers delayed jobs, assigns work to a pool of fallible workers, and monitors execution to completion.

Requirements

Functional Requirements

  • Delayed Execution: Clients can schedule jobs to run immediately or at a specific future timestamp T (or after a relative delay).
  • Job Management: Clients can inspect job status (QUEUED, RUNNING, COMPLETED, FAILED, CANCELLED) and cancel scheduled or active jobs.
  • Failure Handling: Automatic retries with exponential backoff on transient job failure or worker crash.
  • Timeouts: Enforce maximum execution runtimes; jobs exceeding their timeout must be aborted and marked failed.

Non-Functional Requirements

  • Execution Semantics: At-least-once execution guarantee (downstream tasks are expected to be idempotent).
  • Precision & Timeliness: Delayed jobs should be dispatched within 1 second of their scheduled execution time.
  • High Availability & Fault Tolerance: No single point of failure; crashes of scheduler nodes or worker nodes must not cause orphaned or permanently stuck jobs.
  • Multi-tenant Fairness: Prevent a single tenant from starving the worker pool.

Scale Assumptions

  • 10 million jobs submitted per day (~120 submissions/sec average, 1,000/sec peak).
  • Up to 2 million delayed jobs pending in the schedule at any given time.
  • Worker pool consisting of 500 to 2,000 heterogeneous compute nodes.
  • Job runtimes range from 1 second to 2 hours.

What to Cover

  1. APIs & Data Model: Core endpoints (SubmitJob, GetStatus, CancelJob) and persistent schema tracking job states, retry counts, and execution leases.
  2. Architecture: End-to-end design showing how jobs transition from submission, to the delay/scheduling layer, to the dispatch queue, and finally to workers.
  3. Worker Liveness & Lease Management: How the scheduler detects crashed workers and reassigns tasks without causing unbounded duplicate execution.
  4. Cancellation & Race Conditions: Handling the race condition where a job is cancelled right as a worker begins processing it.

Hints

Hint 1

Consider decoupling the delayed timer storage (such as a sorted set or partitioned database index on `scheduled_time`) from the active execution message queue.

Hint 2

Think about using lease-based ownership or visibility timeouts with periodic worker heartbeats to detect and recover from worker node crashes.

Hint 3

For cancellation races, assign an incrementing execution epoch or version token to each run attempt so workers can verify authorization during heartbeats and task completion.

Follow-up questions

  • How would you extend this system to support directed acyclic graph (DAG) job dependencies where Job B depends on the output of Job A?
  • If a single tenant submits millions of short jobs at once, how would you enforce fair queueing and backpressure to protect other tenants?
  • How does your design address clock skew across scheduler instances if jobs require sub-second trigger accuracy?