Katkat
Back

Design CI/CD Build Caching

OpenAISoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This problem evaluates your ability to design a distributed caching system for CI/CD pipelines, focusing on cache key strategies, invalidation, and storage scalability.

System Design: CI/CD Build Cache

You are tasked with designing a distributed cache service for a high-concurrency CI/CD platform. The system supports thousands of concurrent builds per minute across various languages (Python, Go, C++). Your goal is to reduce build times by storing and retrieving intermediate artifacts (e.g., node_modules, compiled object files, container layers) so that they do not need to be rebuilt from scratch.

Functional Requirements

  • Save/Retrieve: Workers must be able to upload cache bundles after a successful build step and download them before execution.
  • Cache Keying: The system must determine if a cache hit exists based on input sources (source code hash, environment variables, lock files).
  • Eviction: Support basic TTL-based eviction to manage storage costs.

Non-Functional Requirements

  • Latency: Retrieving a cache entry must be significantly faster than rebuilding from source.
  • Availability: The cache service should not block the build process; if the cache is down, builds must still proceed (fall back to clean builds).
  • Concurrency: Multiple workers may attempt to write the same cache key simultaneously (the "thundering herd" problem).

Requirements for your Design

  1. API Interface: Define the endpoints for interacting with the cache service.
  2. Storage Model: Propose a storage architecture that balances high read throughput and durability.
  3. Key Strategy: Explain how you would generate unique, deterministic keys to ensure cache correctness.
  4. Consistency: Describe how you handle race conditions when two identical builds finish at the same time and attempt to upload the same artifact.

Hints

Hint 1

Consider using an object store like S3 for the actual large binary artifacts and a fast metadata store like Redis to map cache keys to object store locations.

Hint 2

A deterministic cache key should be a hash of all inputs that affect the build output, including dependency lock files, compiler versions, and relevant build flags.

Hint 3

Use a 'lock-then-upload' or 'temporary-staging' strategy to handle the case where multiple workers attempt to write the same key simultaneously.

Follow-up questions

  • How would you implement tenant isolation to ensure that a malicious build cannot access cache entries belonging to another project?
  • If the network bandwidth between your build workers and the cache storage becomes the bottleneck, what strategies would you use to mitigate it?
  • How would you handle cache 'poisoning', where a broken build artifact is mistakenly cached and reused by future builds?