Katkat
Back

Design a Reliable Job Scheduler

SnowflakeSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This problem tests the ability to design a distributed, fault-tolerant job scheduler capable of handling task persistence, concurrency, and strict timing requirements at scale.

Problem Statement

Design a distributed job scheduler that supports one-time jobs (run at timestamp T) and recurring jobs (run every X seconds). The system must guarantee at least-once execution while minimizing duplicate execution in a multi-worker environment.

Functional Requirements

  1. ScheduleJob(job_id, payload, run_at_timestamp): Persist a job to be executed.
  2. ScheduleRecurringJob(job_id, payload, interval): Persist a job that triggers repeatedly.
  3. ExecuteJobs(): Workers pull and run jobs that are due.

Non-Functional Requirements

  • Scale: Support millions of active scheduled jobs.
  • Fault Tolerance: If a worker crashes while processing a job, the job must be picked up by another worker.
  • Precision: Jobs should trigger within a few seconds of their run_at_timestamp.
  • Performance: Minimize latency between a job becoming due and being picked up.

Constraints

  • Use a database of your choice for persistence.
  • Assume you have N worker nodes.
  • Consider clock skew across distributed servers.

Expected Output

Your design should address:

  1. Data Model: Schema design for storing job states and schedules.
  2. Job Acquisition: How workers claim jobs without conflicting with others (e.g., locking strategies or sharding).
  3. Failure Handling: How to detect worker death and ensure jobs are re-queued.
  4. System Architecture: High-level components (API layer, Job Store, Polling/Scheduling logic, Execution nodes).

Hints

Hint 1

Consider using a 'Status' field (e.g., PENDING, CLAIMED, COMPLETED) combined with a TTL-based lease to prevent race conditions during job acquisition.

Hint 2

Instead of every worker querying the entire table, evaluate if partitioning the job store by a hash of the JobID or timestamp ranges improves scalability.

Hint 3

For high-frequency jobs, differentiate between a 'Trigger' process that moves jobs to a work queue and the 'Execution' process that performs the actual task.

Follow-up questions

  • How would you handle a scenario where a specific job takes longer to execute than the lease duration provided by the scheduler?
  • If the database becomes a bottleneck during peak polling times, what caching or architectural changes would you implement?
  • How do you ensure idempotency if your system architecture accidentally triggers the same task twice during a failover?