Katkat
Back

Design a distributed job scheduler service

AmazonSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed, fault-tolerant system capable of scheduling and executing tasks at a massive scale with strict latency requirements.

Problem Statement

Design a distributed job scheduler service that allows clients to submit tasks to be executed at a specific future time or on a recurring schedule.

Functional Requirements

  • Schedule: Support one-time delayed jobs and cron-like recurring jobs.
  • Persistence: Jobs must be persisted and survive service crashes.
  • Execution Guarantee: Each job should be triggered at or after its scheduled time.
  • Scalability: The system must handle millions of scheduled jobs and thousands of executions per second.

Non-Functional Requirements

  • Reliability: At-least-once execution; minimized latency between scheduled time and actual execution.
  • Availability: The system should remain functional even if individual nodes fail.

Deliverables

  1. High-level architecture diagram (described in text).
  2. API interface definitions for job management (Create, Update, Cancel).
  3. Database schema design and storage strategy.
  4. Strategy for handling high-volume job triggers and worker assignment.
  5. Discussion on how to handle worker node failures and duplicate executions.

Hints

Hint 1

Consider using a distributed queue or a specialized sorted structure to track jobs ordered by their next execution timestamp.

Hint 2

Think about the partitioning strategy for the database or storage layer to avoid contention on the 'next job' lookup as the number of jobs grows.

Hint 3

Use a coordination service like ZooKeeper or etcd for leader election or task partitioning if multiple scheduler nodes are running simultaneously.

Follow-up questions

  • How would you handle a scenario where a specific job takes significantly longer to execute than the interval at which it is scheduled?
  • How can you implement backpressure to prevent the scheduler from overwhelming worker nodes during spikes in task volume?
  • What changes would you make if the jobs required global ordering (i.e., Job B must start only after Job A completes successfully)?