Katkat
Back

Design a distributed job scheduler

RobinhoodSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a fault-tolerant, scalable distributed system capable of scheduling one-time and recurring background tasks with guaranteed execution semantics.

Problem Statement

Design a distributed job scheduler service that allows users to schedule background tasks. The system must support two types of jobs:

  1. One-time jobs: Executed at a specific future timestamp.
  2. Recurring jobs: Executed based on a cron-like schedule (e.g., "every Monday at 8 AM").

Functional Requirements

  • Users can create, delete, and view the status of their jobs.
  • The system must ensure 'at-least-once' execution guarantees.
  • Support task prioritization (e.g., high-priority jobs should be picked up before low-priority ones).
  • The system should handle failures gracefully (e.g., if a worker node crashes during job execution).

Non-Functional Requirements

  • Scale to handle 100,000 scheduled jobs per minute.
  • Support low latency for job submission and status retrieval.
  • High availability: the scheduler should not have a single point of failure.

Deliverables

  • Describe the high-level system architecture and how services interact.
  • Define the database schema for storing jobs and their execution states.
  • Explain the mechanism for picking up jobs that are due for execution.
  • Discuss how the system handles worker failures and retries for failed task executions.

Hints

Hint 1

Consider using a distributed queue or a polling mechanism from a database with a timestamp index to identify ready-to-run jobs.

Hint 2

Think about how to implement a locking mechanism (e.g., optimistic locking or distributed locks) to ensure that two worker nodes do not pick up the same job simultaneously.

Hint 3

Use a state machine for job statuses, such as PENDING, RUNNING, COMPLETED, and FAILED, to track the lifecycle of each task.

Follow-up questions

  • How would you handle backpressure if the number of jobs scheduled for a specific time exceeds the current worker capacity?
  • How would you implement 'exactly-once' execution semantics if the task side-effects are not idempotent?
  • What metrics and monitoring tools would you implement to identify stalled jobs or system bottlenecks?