Katkat
Back

Design a CI/CD system with stuck-job handling

OpenAISoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This problem tests your ability to design a distributed job scheduling and orchestration system that handles process failures, network partitions, and job timeouts to ensure reliable execution in a high-scale environment.

Problem Statement

Design a CI/CD pipeline orchestration system capable of managing thousands of concurrent jobs across a fleet of worker nodes. A critical requirement is handling 'stuck jobs'—jobs that have been claimed by a worker but are no longer making progress or have lost heartbeat connectivity.

Requirements

  1. Job Lifecycle: Jobs move through states: PENDING, RUNNING, COMPLETED, FAILED, and TIMED_OUT.
  2. Fault Tolerance: The system must detect when a worker node dies or a network partition occurs, preventing jobs from being 'lost' or stuck in the RUNNING state indefinitely.
  3. Concurrency & Multi-tenancy: The system must support multiple users with different priority levels and resource quotas.
  4. Scalability: Design for a system capable of handling 10,000+ jobs per hour.

Output

Your design should cover:

  • Data Model: The schema for jobs, worker heartbeats, and state transitions.
  • Failure Detection: How to distinguish between a slow job and a dead worker/process.
  • Re-queuing Mechanism: How to safely transition a stuck job back to the queue without causing race conditions or duplicate execution side-effects (e.g., duplicate deployments).
  • API Interfaces: Key endpoints for job submission, status polling, and worker heartbeat updates.

Constraints

  • Assume an asynchronous task queue architecture (e.g., using Redis or Kafka).
  • Consider the trade-offs between consistency and availability in your failure detection strategy.

Hints

Hint 1

Use a distributed lease mechanism where workers must periodically renew their ownership of a job via a heartbeat timestamp in the database.

Hint 2

Implement an 'observer' service that scans the database for jobs with heartbeat timestamps older than a specific threshold to identify candidates for re-queuing.

Hint 3

Introduce idempotency tokens or generation counters for jobs to ensure that if a stale job finally completes, its results are ignored if a new instance of that job has already started.

Follow-up questions

  • How would you handle a 'poison pill' job that consistently crashes the worker that picks it up?
  • If your system relies on a database for state, how do you prevent the heartbeat updates from overwhelming the database at high scale?
  • How would you implement a 'cancel' feature that works even if the worker node is under heavy CPU load or unresponsive?