Katkat
Back

Design a Distributed Job Scheduler with Cron, Pause and Resume

SnowflakeSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a job scheduler service that runs one-time and cron jobs, supports pausing and resuming, and dispatches work to an unlimited pool of workers. It tests due-job indexing, claim and lease protocols, pause semantics, at-least-once execution with idempotency, and handling top-of-the-hour spikes.

Design a job scheduler service. Users register jobs that run once at a given time or repeatedly on a cron schedule, and they can pause a job and later resume it. Assume an unlimited pool of workers is available to execute jobs, so the design is about deciding when each job runs, dispatching it exactly when it is due, and keeping track of every run.

Constraints and Clarifications

  • Given: cron schedules, one-time jobs, pause and resume, and unlimited worker capacity.
  • Not given: the number of jobs, how precisely a job must start on time, how long jobs run, or what guarantee a run must have (at least once or exactly once). Ask, or state your assumptions.

Clarifying Questions

  • How many jobs are registered, and how many fire at the busiest moment, such as the top of the hour?
  • How close to its scheduled time must a job start: within a second, or within a minute?
  • Is it acceptable for a run to happen twice after a failure, or must each scheduled occurrence run exactly once?
  • If a run is still going when the next occurrence is due, should the next one start, wait or be skipped?
  • What does pausing do to a run that is already executing, and on resume, should occurrences missed during the pause run?
  • In which time zone are cron expressions evaluated, and how should daylight-saving changes behave?

Hints

Hint 1

Find the due jobs cheaply: Think about how the scheduler finds the jobs due in the next second without scanning every registered job, and what it stores to make that lookup fast.

Hint 2

One run, one owner: Several scheduler instances and retrying workers can all touch the same run. Decide what identifies a run uniquely and how ownership of it is claimed and released.

Follow-up questions

  • Ten million jobs are scheduled for midnight. How do you keep that moment from overwhelming the scheduler and the downstream systems?
  • A worker dies halfway through a run. How is that detected, and what happens to the run?
  • A job runs at 02:30 local time every day. What happens on the days the clocks change?
  • Users want job B to start only after job A succeeds. How do you add dependencies?