Katkat
Back

Design a CI/CD system with live log streaming

OpenAISoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a distributed job orchestration system that handles task dependencies, secure execution, and low-latency log streaming at scale.

System Design: CI/CD Pipeline Orchestrator

Design a CI/CD service that allows developers to define pipelines as a Directed Acyclic Graph (DAG) of jobs. Each job runs as a containerized script. The system must support near-real-time streaming of job logs to a web dashboard, persistence of job history, and secure secret management.

Functional Requirements

  1. Pipeline Execution: Execute jobs based on DAG dependencies (e.g., jobs run only if dependencies succeed).
  2. Log Streaming: Users must see job logs in real-time as the container generates stdout/stderr.
  3. Persistence: Store job logs and metadata (status, exit code, duration) for at least 30 days.
  4. Security: Jobs must have isolated environments; secrets (API keys) must be injected securely without appearing in logs.

Non-Functional Requirements

  1. Scale: Support 1,000 concurrent pipelines across thousands of nodes.
  2. Low Latency: Log streaming delay should be under 500ms.
  3. Resiliency: The system must handle agent crashes and network partitions gracefully.

What to include in your design

  • API Design: Define the interface for triggering pipelines and fetching logs.
  • Data Model: Schema for tracking jobs, DAGs, and log pointers.
  • Architecture: How the orchestrator schedules jobs on workers, how the worker captures logs, and the path logs take to the user.
  • Trade-offs: Explain your choices regarding log storage (e.g., object storage vs. databases) and communication protocols (e.g., WebSockets vs. gRPC streaming).

Hints

Hint 1

Consider using a sidecar pattern within the execution container to capture and push log streams to a message bus.

Hint 2

Decouple the job scheduling logic (the brain) from the execution environment (the muscle) using a message queue or persistent task store.

Hint 3

For log persistence, use a two-tiered strategy: high-throughput stream ingestion for live viewing and batch flushing to cold storage for long-term audit.

Follow-up questions

  • How would you implement 'exactly-once' execution semantics if an agent node fails mid-job?
  • If logs reach 50GB per pipeline, how do you optimize log storage to keep the system performant?
  • How would you implement a rate-limiting mechanism to prevent a user from accidentally spinning up thousands of jobs simultaneously?