Katkat
Back

Design a Distributed Cron Job Scheduler

AmazonSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a fault-tolerant, horizontally scalable distributed cron scheduler capable of triggering millions of recurring tasks with at-least-once delivery semantics and deterministic run coordination.

Scenario

Many cloud platforms and enterprise architectures require a centralized distributed cron service capable of triggering recurring tasks (e.g., periodic database backups, billing rollups, and metric aggregations). Design a highly available, multi-tenant distributed cron scheduler that can schedule, dispatch, and track recurring jobs with resilience against coordinator crashes, network partitions, and slow downstreams.

Requirements

Functional Requirements

  • Schedule Management: Users can create, update, pause, delete, and manually trigger recurring schedules with standard cron expressions and explicit time zone specifications.
  • Execution Dispatch: The system triggers runs at their target times and dispatches execution payloads to remote target endpoints (e.g., HTTP webhooks, AWS SQS queues, or internal worker nodes).
  • Lifecycle & Retries: Track execution attempts, enforce timeouts, handle exponential backoff retries, and support in-flight run cancellations.
  • Backfilling & Catch-up: Allow users to run backfills for past windows or configure policies for missed triggers (e.g., fire once vs. catch up sequentially).

Non-Functional Requirements

  • Scale: Support 50 million registered jobs with peak dispatch rates of up to 50,000 runs per second.
  • Timeliness: 99% of tasks must be dispatched within 1 second of their scheduled trigger time.
  • Fault Tolerance & Consistency: Guarantee at-least-once job triggering despite node failures, split-brain scenarios, or database failovers. Avoid duplicate execution where possible via deterministic run IDs.
  • Multi-tenant Fairness: Prevent high-frequency or bursty jobs from starving lower-frequency workloads.

Deliverables

  1. High-Level Architecture: Core components (API service, scheduler/dispatcher nodes, storage layer, execution queue, and worker pool) and their interactions.
  2. Data Model: Schemas for job definitions, scheduled run instances, and attempt logs.
  3. Scheduling & Partitioning Strategy: How schedules are scanned, partitioned across scheduler nodes, and dispatched without contention or duplicate runs.
  4. Failure Modes & Edge Cases: Handling lease expirations, worker crashes, cancellation races, time-zone transitions (e.g., daylight saving time), and reconciliation of orphaned runs.

Hints

Hint 1

Generate deterministic run IDs (such as combining `job_id` and the truncated scheduled timestamp in UTC) to enforce idempotency at the database or queue layer.

Hint 2

Avoid scanning the entire job database every minute; consider partitioning jobs by trigger time into discrete time buckets or using a distributed delay queue / priority wheel.

Hint 3

Use short-lived distributed leases with heartbeats for active runs, coupled with a secondary reconciliation loop to detect and reassign abandoned executions.

Follow-up questions

  • How does your design handle Daylight Saving Time (DST) shifts where a wall-clock hour is repeated or skipped entirely?
  • If a single tenant schedules 100,000 tasks at the top of the hour, how do you prevent worker pool starvation and enforce fair sharing across tenants?
  • How would you handle downstream targets that cannot provide idempotent receivers, minimizing the window of duplicate execution during worker failover?