Design a Durable Task Scheduling Service

Scenario

Many machine learning and data labeling workflows require scheduling actions in the future: triggering hourly model evaluation benchmarks, kicking off overnight data sync pipelines, or retrying failed webhook deliveries with exponential backoff.

Design a multi-tenant, durable distributed task scheduling service that allows client systems to schedule, cancel, and monitor delayed one-time tasks and recurring (cron) tasks, while strictly decoupling the scheduling coordination from actual job execution.

Requirements

Functional Requirements

• Schedule Tasks: Support one-time tasks scheduled at an absolute timestamp (run_at) and recurring tasks defined by a cron expression (schedule).
• Decoupled Execution: The scheduler should be responsible only for initiating the task execution (e.g., dispatching an event to a message queue or invoking an execution worker), not for the task computation itself.
• Cancellation & Updates: Allow callers to cancel or reschedule pending tasks before their execution begins.
• Execution Guarantees: Ensure at-least-once dispatch semantics. Downstream consumers should be provided with mechanisms (like idempotency keys) to prevent duplicate runs.
• Retry & Recovery: If task dispatch fails or an executor crashes mid-run, support configurable retries with backoff and eventual dead-letter routing.

Non-Functional Requirements

• Scale: Maintain up to 50 million pending tasks at any time; sustain a peak dispatch rate of 10,000 tasks/second.
• Accuracy & Latency: Tasks should be dispatched within 1 second of their scheduled trigger time.
• Durability: Zero task loss once acknowledged. State must persist across node crashes and cluster re-elections.
• Fault Tolerance: Gracefully handle clock drift, node failures, and high concurrency on identical execution targets.

What Your Design Should Cover

1. API & Data Model: Define endpoints for scheduling, cancellation, and status lookups. Design schema for task definitions, schedules, and run history.
2. Time-Wheel / Due-Work Claiming: How due tasks are discovered, partitioned, and claimed across multiple scheduler nodes without contention or duplicate dispatches.
3. Decoupled Dispatch: How the scheduler hands off work to execution runtimes (pull vs. push, message queues, leases).
4. Failure Modes & Edge Cases: Handling thundering herds (e.g., 100k tasks scheduled for midnight), missed runs during scheduler downtime, and recurring task catch-up policies.