Katkat
Back

Design a pipeline orchestration system on Kubernetes

SalesforceSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a container-orchestrated workflow system that manages dependencies, task execution, and system reliability.

Problem Statement

Design a distributed pipeline orchestration system that runs on top of Kubernetes. Users should be able to submit workflows composed of multiple tasks, where each task is a containerized job. These tasks often have dependencies, forming a Directed Acyclic Graph (DAG).

Functional Requirements

  • Workflow Submission: Users submit a JSON definition of a DAG (nodes as containers, edges as dependencies).
  • Task Execution: Tasks must execute in order. A task starts only when all its parent tasks have succeeded.
  • Failure Handling: Support automatic retries and custom backoff policies for individual tasks.
  • State Persistence: The system must track the state of the workflow and individual tasks so it can resume after a control-plane restart.

Non-Functional Requirements

  • Scalability: The system should support thousands of concurrent workflow runs.
  • Isolation: Workflows from different tenants must be logically isolated.
  • Observability: Users need to see logs and real-time status of each task.

Design Deliverables

  • Define the API for submitting and monitoring workflows.
  • Describe the architecture, specifically how you use Kubernetes primitives (e.g., Custom Resource Definitions, Controllers, Jobs).
  • Explain how task dependency resolution and state management are handled.
  • Discuss how you ensure system reliability if the orchestrator pod crashes.

Hints

Hint 1

Consider using Kubernetes Custom Resource Definitions (CRDs) to represent 'Workflow' and 'Task' objects within the cluster.

Hint 2

Think about whether to use a central controller loop (like the Kubernetes reconciliation pattern) or a sidecar-based approach for tracking task completion.

Hint 3

Address how you would store state; a database (e.g., PostgreSQL) or an internal K8s status field are common choices.

Follow-up questions

  • How would you handle tasks that require persistent storage (e.g., intermediate data passed between tasks)?
  • How would you implement a concurrency limit so that one user doesn't saturate the entire cluster with their tasks?
  • If a task is stuck (e.g., node goes down), how does your system detect it and trigger a recovery?