Katkat
Back

Design Large-Scale Inference Serving

WaymoMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenML System Design

Last updated

This question evaluates understanding of large-scale ML inference systems, assessing competencies in capacity planning, latency and tail-latency engineering, memory and bandwidth estimation, hardware selection (CPUs/GPUs/specialized accelerators), batching and caching trade-offs, and reliability concerns such as out-of-memory prevention and recovery. It is commonly asked to test practical system-design skills for production deployment by requiring back-of-the-envelope QPS and resource estimates and reasoning about operational trade-offs; this belongs to the ML system design category and emphasizes practical application over purely conceptual theory.

Design a production inference serving system for a machine learning model used by 100 million daily active users. Your answer should cover: traffic assumptions and back-of-the-envelope QPS estimates; memory requirements for model weights, activations, caches, and batching; network and accelerator bandwidth estimates; how to choose CPUs, GPUs, or specialized accelerators; how to optimize latency and tail latency; and how to prevent or recover from out-of-memory failures.