Design a cloud AI inference platform

HardSoftware Engineer HR Screen ML System Design

Last updated October 27, 2026

This question evaluates competency in ML system design and operational engineering, covering model packaging and versioning, hardware selection (CPU vs GPU), autoscaling, request routing, multi-tenant isolation, SLO-driven observability, cost controls, and failure handling within a cloud inference platform.

Design a cloud-based AI inference platform for real-time and batch workloads. Specify model packaging and versioning, hardware selection (CPU vs. GPU), autoscaling, request routing, and multi-tenant isolation. Discuss latency/throughput targets, cost controls (right-sizing, spot instances), observability (tracing, metrics), safe rollout strategies (canary, shadow), and failure handling. Describe how you would integrate with Kubernetes and a model registry, and compare your approach to similar industry offerings.

Design and explain infra, K8s, routing, and fleetHard
Lambda · System Design · HR Screen
Add an LLM-backed recommendation API to an existing service with RAG and promptingMedium
Intuit · ML System Design · Onsite
Applying LLMs to Autonomous Driving Despite Hallucination and Real-Time LimitsHard
Aurora · ML System Design · Technical Screen
Architect an asynchronous RL post-training systemHard
Meta · ML System Design · Onsite
Build a Candidate Search SystemMedium
Mercor · ML System Design · Onsite
Build a Mistral-powered RAG agentHard
Meta · ML System Design · Onsite