Katkat
Back

Design a High-Frequency Job Scheduler

ByteDanceSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests your ability to design a distributed, high-frequency job scheduling system that handles millions of tasks with varying priorities while ensuring reliability and strict execution guarantees.

System Design: High-Frequency Job Scheduler

Design a distributed job scheduling system capable of managing millions of unique tasks. The system must support "one-off" scheduled jobs (e.g., "execute at 10:00 AM") and recurring jobs (e.g., "execute every 5 minutes").

Requirements

  • Functional:
    • Ability to enqueue, update, and cancel jobs.
    • Support for job priorities (High, Medium, Low).
    • Execution guarantees: Ensure that a task is executed at least once.
    • Idempotency support: Allow jobs to define an idempotency key to prevent accidental double-execution.
  • Non-Functional:
    • High Scalability: The system should handle 10,000+ job triggers per second.
    • Low Latency: Once a job's scheduled time arrives, it should be dispatched within 100ms.
    • Fault Tolerance: If a worker node crashes mid-execution, the job must be recoverable.

Expectations

Your design should cover:

  1. Data Model: How to represent jobs, job definitions, and execution states.
  2. Architecture: Components for job ingestion, storage, scheduling (timing), and worker dispatching.
  3. Scaling: How to handle peak load when millions of jobs trigger at the same time.
  4. Failure Handling: Strategies for handling worker timeouts, crashes, and network partitions.

Provide an API signature for scheduleJob(job_details) and cancelJob(job_id).

Hints

Hint 1

Consider using a combination of a persistent database for long-term storage and a high-performance in-memory structure (like a Time-Wheel or a Priority Queue) for pending job dispatching.

Hint 2

Decouple the scheduler (the clock) from the execution (the worker pool) using a message queue to prevent the scheduler from being overwhelmed by execution latency.

Hint 3

To handle failures, implement a 'lease' or 'visibility timeout' mechanism where workers must claim a job and periodically renew their ownership until completion.

Follow-up questions

  • How would you handle a 'thundering herd' problem if 1 million jobs are scheduled for the exact same second?
  • How would you ensure that a recurring job does not overlap with its own previous instance if it takes longer than expected to execute?
  • If your system relies on a central database, how would you partition or shard the database to prevent it from becoming a bottleneck as the job count grows?