Katkat
Back

Design a CI Job Scheduler with Live Build Output

LinkedInDevOps Engineer · Onsite
MediumDevOps EngineerOnsiteSystem Design

Last updated

Design a scalable continuous integration (CI) platform that schedules dependency-aware build jobs across worker fleets and streams live console output to end users.

Scenario

Modern engineering organizations rely on automated continuous integration (CI) platforms to validate code changes before deployment. You are tasked with designing the core backend for an enterprise CI engine (similar to LinkedIn's internal CI infrastructure). The system must ingest code change events, resolve job dependency graphs (DAGs), dispatch tasks to worker fleets, and provide developers with sub-second, real-time streaming logs of build output.

Requirements

Functional Requirements

  • Triggering & Deduplication: Ingest VCS events (webhooks) reliably. Support build deduplication (e.g., auto-canceling superseded builds on the same branch).
  • Dependency-Aware Scheduling: Model pipeline steps as a Directed Acyclic Graph (DAG) and dispatch jobs as upstream dependencies succeed.
  • Worker Execution & Leases: Dispatch jobs to isolated worker pools (VMs or containers). Implement worker heartbeats and leases to detect worker death and retry failed jobs.
  • Live Streaming & Archival: Stream stdout/stderr from running containers to active web sessions with minimal latency; persist final logs to durable cold storage upon job completion.
  • Fairness & Quotas: Prevent large teams or runaway jobs from starving smaller tenants of worker capacity.

Non-Functional Requirements

  • Reliability: At-least-once execution for jobs; exactly-once state transitions for pipeline stages.
  • Low Latency: Queue-to-worker dispatch time under 5 seconds when capacity is available; log streaming latency under 1 second from worker emit to web UI.
  • Scalability: Horizontal scaling for both the scheduling control plane and log streaming data path.

Scale Assumptions

  • Pipeline Volume: 50,000 pipeline executions per day.
  • Concurrency: Up to 10,000 concurrently running jobs during peak hours.
  • Log Throughput: Average build generates 5 MB of logs across 10 minutes. Peak aggregate log ingestion rate is ~200 MB/s.
  • Log Retention: Hot/live logs retained for 7 days; compressed cold archive stored for 90 days.

Key Areas to Cover

  1. Architecture & Ingestion: Ingestion gateway (webhook vs. polling), event deduplication, and pipeline DAG state representation.
  2. Scheduler & Lease Management: Task queue design, worker leasing mechanism, heartbeat timeouts, and safe recovery of stranded tasks.
  3. Live Output Pipeline: Log aggregation architecture from worker daemon to browser clients (e.g., Redis pub/sub, Kafka, WebSockets), and subsequent flushing to object storage (e.g., S3/Blob storage).
  4. Isolation & Quotas: Worker container isolation, resource caps, and fair-share queueing strategies across multiple organizations.

Hints

Hint 1

Model the build pipeline as a DAG where task nodes transition from PENDING to READY once all direct parent nodes reach the SUCCESS state.

Hint 2

Decouple live log streaming from persistent storage: stream raw chunks via an append log or message bus (like Kafka) to WebSocket servers, while a background flusher writes batched segments to object storage.

Hint 3

Use distributed short-lived leases with worker heartbeats to detect hung runners, ensuring idempotent retries and preventing duplicate concurrent executions.

Follow-up questions

  • How would you enforce fair-share worker allocation when a single massive repository pushes dozens of rapid commits and threatens to starve other teams?
  • If a build step hangs indefinitely or a developer manually clicks 'Cancel Build', how does the cancellation signal promptly and cleanly propagate down to the sandboxed worker process?
  • How would you handle network partitions where a worker continues executing a task after its lease has expired and the scheduler has already re-assigned the task to another runner?