Design a Task Scheduler and Executor

Design a service that accepts tasks for future or recurring execution and dispatches them to workers.

Constraints & Assumptions

• Practice assumption: tasks include one-time, fixed-delay, and cron-like schedules.
• Cron and fixed-rate schedules advance from their intended occurrence when dispatch is durably created. A fixed-delay schedule creates its next occurrence only after the current occurrence reaches its configured terminal completion state; retries remain part of the same occurrence.
• At peak, clients create 100000 tasks per second and 100 million tasks may be pending.
• Execution is at least once; clients can supply an idempotency key.
• A task may run for seconds or hours and can be retried.
• Dispatch within one second of the due time is acceptable under healthy conditions.

Clarifying Questions to Ask

• Is exactly-once business effect required, or only exactly-once scheduling intent?
• How are task payloads stored, secured, and versioned?
• What retry, cancellation, priority, and tenant-isolation semantics are required?
• Can recurring executions overlap?
• For fixed-delay work, does the delay begin after success, any terminal result, or final retry exhaustion?
• How should cron time zones, daylight-saving transitions, and missed occurrences behave?

Part 1: API and Data Model

Define creation, update, cancellation, inspection, and execution-attempt records. Explain state transitions and concurrency control.

Hints

• A logical task and one attempt are not the same entity.

What This Part Should Cover

• Stable identifiers and idempotency
• Explicit states and versions
• Schedule representation
• Distinct advancement rules for cron, fixed-rate, and fixed-delay schedules

Part 2: Finding and Dispatching Due Work

Explain how schedulers find due tasks without scanning the full table, avoid duplicate ownership, and feed executors with backpressure.

Hints

• Consider partitioning by both time and ownership.

What This Part Should Cover

• Due-time indexing
• Safe concurrent claiming
• Queueing and overload behavior

Part 3: Execution, Retry, and Recovery

Describe worker leases, heartbeats, retry policy, poison tasks, cancellation races, and recovery after scheduler or worker failure.

Hints

• Separate acknowledgment of dispatch from completion of business work.

What This Part Should Cover

• At-least-once consequences
• Bounded retry and dead-letter handling
• Idempotent creation of the next fixed-delay occurrence after terminal completion
• Observability and reconciliation