Katkat
Back

Design a CI system that runs one million tests in ten minutes

GoogleSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a continuous integration system that builds a change and runs about one million tests within a ten-minute budget. Tests capacity estimation, sharding and scheduling by test duration, artifact distribution, caching and test selection, flaky tests, failure handling, and cost versus latency trade-offs.

Design a continuous integration (CI) system for a large codebase whose test suite has about 1,000,000 tests. When a developer submits a change, the system must build it, run the tests and report which tests failed, and a full run of the suite must complete within 10 minutes. Explain how the design meets that budget, and argue for each choice through its trade-offs rather than by listing components.

Constraints and Clarifications

  • About 1,000,000 tests per full run.
  • A full run must finish within 10 minutes of wall-clock time.
  • Test durations, change volume and the cost budget are not given. Ask for them, or state your assumptions explicitly.

Clarifying Questions

  • Does the 10-minute budget include building the code and getting machines ready, or only running the tests?
  • What do test durations look like: mostly sub-second unit tests, or a long tail of multi-minute integration tests? Is per-test duration history available?
  • Must every change run all 1,000,000 tests, or may the system run only the tests a change can affect?
  • Are tests hermetic (no shared state, no network calls), and how often are tests flaky?
  • How many changes are submitted per hour at peak, and is there a limit on machine cost?

Hints

Hint 1

Do the arithmetic first: Before drawing any boxes, divide the total test time by the budget. The number of parallel workers this implies, and the duration of the single slowest test, drive most of the later decisions.

Hint 2

What sets the finish time: When the work is spread over thousands of machines, ask which machine's work decides when the result is ready, and what you can do about it.

Follow-up questions

  • One test alone takes 12 minutes. What do you do?
  • Fifty changes are submitted in the same minute. How does each still finish within 10 minutes without 50 times the machines?
  • How would you detect that a test has become flaky, and what happens to a change it fails?
  • How do you check that test selection or result caching is not silently skipping tests that should have run?