Design a Distributed Job Scheduler with Run Logs and Exactly-Once Execution

Scenario

You are asked to design a scalable distributed job scheduler for a multi-tenant cloud platform (similar to AWS EventBridge combined with AWS Batch). The platform allows users to schedule arbitrary compute tasks, monitors their execution across a dynamic pool of worker machines, and stores detailed execution logs for auditing and debugging.

A critical requirement emphasized by stakeholders is handling failure modes gracefully: workers may crash mid-task, network partitions will occur, and duplicate executions must be strictly controlled or prevented.

Requirements

Functional Requirements

• Job Definition & Scheduling: Support one-off jobs (execute at timestamp T) and recurring jobs (cron-based schedules).
• Job Execution: Assign jobs to a pool of distributed workers based on priority and availability.
• Status & Run Logs: Capture real-time status transitions (e.g., SCHEDULED, RUNNING, COMPLETED, FAILED) and ingest/store output logs (stdout/stderr) for each execution run.
• Cancellation & Retries: Allow users to abort scheduled or running jobs, and configure retry policies with exponential backoff on failure.
• Delivery & Execution Guarantees: Address the client expectation of "exactly-once execution" and design mechanisms (e.g., distributed leases, fencing tokens, idempotency) to handle duplicates and split-brain scenarios.

Non-Functional Requirements

• Scalability: Support 100 million registered job definitions, triggering up to 100,000 executions per minute at peak (~2,000 executions/second).
• Dispatch Latency: Jobs should begin executing within 5 seconds of their scheduled time.
• Log Retention & Durability: Run logs must be durable and retrievable for at least 30 days.
• Fault Tolerance: No single point of failure; scheduler nodes or workers crashing must not result in permanently orphaned or duplicate-executed jobs.

What Your Design Should Cover

1. Core Architecture & Component Flow: Scheduler coordinator, time-bucketed delay queues, worker pools, and heartbeat managers.
2. Data Model & State Management: Schema for job metadata, execution runs, and schedule triggers.
3. Exactly-Once vs. At-Least-Once Analysis: Realistic trade-offs regarding distributed consensus, fencing tokens, and idempotent execution keys.
4. Log Ingestion Pipeline: Architecture for handling bursty log streams from workers without impacting scheduling latency.
5. Edge Cases: Handling worker network isolation, clock skew across nodes, and worker crash during log flushing.