Katkat
Back

Design a Reliable Distributed Job Scheduler

RobinhoodSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a distributed, fault-tolerant job scheduler capable of handling job lifecycle states, worker coordination, and consistent scheduling across multiple tenants.

System Design: Distributed Job Scheduler

Design a distributed job scheduling system that allows users to submit one-time and recurring (cron-like) jobs. The system must ensure that jobs are executed exactly once (or as close as possible) despite worker crashes, network partitions, or system restarts.

Requirements

Functional Requirements:

  1. Job Submission: Clients can schedule jobs with an execution time (T) and a callback endpoint/payload.
  2. Lifecycle Management: Support pause, resume, and cancellation of pending jobs.
  3. Reliability: Failed jobs should support configurable retry policies (exponential backoff).
  4. Fairness: Ensure that one tenant submitting thousands of jobs does not starve other tenants.

Non-Functional Requirements:

  1. Scalability: The system must handle millions of scheduled jobs per day.
  2. Consistency: Jobs must not be executed twice (or minimize the impact of duplicate delivery via idempotent design).
  3. Latency: Jobs should be triggered within seconds of their scheduled time.

Your Task

Describe the high-level architecture of this system. In your design, please cover:

  1. Data Model: How to store jobs, their state (Pending, Running, Completed, Failed), and metadata.
  2. Due-work Discovery: How the system efficiently finds jobs ready to run without scanning the entire database.
  3. Task Distribution: How tasks are assigned to worker nodes and how you handle worker crashes.
  4. Backpressure: How to prevent the system from being overwhelmed if a massive spike in jobs occurs.

Constraints

  • Assume a standard microservices environment with access to a relational database (e.g., PostgreSQL) and a message queue (e.g., Kafka or RabbitMQ).

Hints

Hint 1

Consider using a 'poll and claim' pattern or a time-bucketed sharding approach to avoid database bottlenecks during due-work discovery.

Hint 2

Think about using a state machine per job to track transitions and a heartbeat mechanism or lease-based locking to detect if a worker has died while processing a job.

Hint 3

Address duplicate delivery by implementing a transaction-safe 'outbox' pattern or requiring that downstream consumers implement idempotent job handling.

Follow-up questions

  • How would you handle jobs that span different time zones, specifically regarding daylight savings time transitions?
  • If you needed to support prioritization (e.g., gold-tier vs. free-tier tenants), how would your worker assignment logic change?
  • How would you perform a rolling update of the worker nodes without dropping in-flight jobs or delaying scheduled ones?