Design a CI/CD Task Scheduler

Scenario

Modern engineering organizations rely on a centralized Continuous Integration and Continuous Delivery (CI/CD) platform to build, test, and deploy software. You are tasked with designing the scheduling and execution core of this platform. The system must accept pipeline definitions expressed as Directed Acyclic Graphs (DAGs), match tasks to capable worker nodes, maintain durable execution state, handle node failures gracefully, and support deployment workflows that require strict safety guarantees.

Requirements

Functional Requirements

• DAG Orchestration: Parse and execute tasks based on topological dependencies. Downstream tasks must only run after upstream dependencies succeed.
• Worker Matching & Leasing: Dispatch tasks to workers matching required labels (e.g., os=linux, arch=arm64, gpu=true). Workers acquire tasks via short-lived leases that must be renewed through heartbeats.
• Failure Handling & Retries: Support task-level retry policies (exponential backoff, max retries) and immediate pipeline cancellation on non-recoverable failures.
• Progress Tracking: Provide queryable, real-time pipeline and task statuses (QUEUED, RUNNING, SUCCESS, FAILED, CANCELLED).
• Deployment Safeguards: Provide strict concurrency control (mutual exclusion or serialized execution) for tasks that deploy to shared target environments (e.g., staging, production).

Non-Functional Requirements & Scale

• Pipeline Scale: 50,000 pipelines executed per day, with an average of 10 tasks per pipeline (500,000 tasks/day).
• Concurrency: Up to 5,000 concurrently active workers. Peak scheduling rate of 50 tasks/second.
• Task Duration: Ranging from 30 seconds (linting) to 2 hours (heavy integration tests/builds).
• Reliability: At-least-once task dispatch with end-to-end task idempotency. Scheduler crashes must not corrupt or lose active pipeline state.
• Fairness: Multi-tenant fairness to prevent a single large repository from starving other teams.

What to Cover

1. Data Model & State Machine: Schema for pipelines, tasks, dependencies, and worker leases. Define the lifecycle states of tasks and pipelines.
2. Architecture & Scheduling Loop: How ready tasks are identified, enqueued, matched to worker capabilities, and dispatched.
3. Worker Protocol: Lease renewal, heartbeat timeouts, and handling of network partitions or zombie workers.
4. Safeguards & Fairness: Mechanisms for environment concurrency limits and tenant queue throttling.