ML System Design interview questions

Showing 31–60 of 332 results

• Design an enterprise RAG agent systemMedium

  This question evaluates a candidate's competency in ML system design for enterprise retrieval-augmented generation and agent architectures, covering understanding of model roles, agent tool integrations, memory and state design, evaluation metrics, security and privacy defenses, concurrency and reliability, workflow durability, and document ingestion/indexing challenges. It is commonly asked in ML system design interviews to assess architectural thinking and trade-off analysis, testing both conceptual understanding and practical application skills for building scalable, secure, and cost-effective enterprise AI assistants.

  AmericanMachine Learning EngineerML System DesignTechnical Screen
• Design GPU inference request batchingMedium

  This question evaluates a candidate's competency in ML System Design, focusing on GPU-based inference batching, scheduling, autoscaling, and the operational concerns of routing, compatibility, and fault-tolerance.

  AnthropicSoftware EngineerML System DesignOnsite
• Design Model Weight DistributionMedium

  This question evaluates a candidate's competency in large-scale ML system design and distributed systems, focusing on weight file distribution, versioning and rollout strategies, consistency models, integrity verification, access control, rollback mechanisms, and operational scalability.

  AnthropicSoftware EngineerML System DesignOnsite
• Design Safe Distribution and Activation of Model WeightsMedium

  This problem tests the ability to architect a robust, secure, and atomic deployment system for large binary artifacts (model weights) across a distributed inference cluster.

  AnthropicSoftware EngineerML System DesignOnsite
• Design a GPU inference APIHard

  This Anthropic ML system design onsite asks you to design a multi-tenant, GPU-backed inference API that serves many models and versions -- including large autoregressive LLMs -- under explicit latency SLOs. It spans API and schema design, streaming and async jobs, continuous batching with prefill/decode separation, KV-cache and GPU memory management, multi-model hosting on shared GPUs, heterogeneous autoscaling and placement, multi-region routing, canary rollout, and reliability and cost controls. The signature probe is the diagnostic scenario where CPU utilization is low but the GPUs are saturated.

  AnthropicSoftware EngineerML System DesignOnsite
• Design a Low-Latency GPU Inference ServiceMedium

  This problem tests the ability to design a high-throughput, low-latency inference architecture for large language models, focusing on hardware utilization and bottleneck mitigation.

  AnthropicSoftware EngineerML System DesignTechnical Screen
• Design a batch inference APIHard

  This question evaluates a candidate's ability to design asynchronous batch inference APIs and systems, including API schema and job lifecycle design, idempotency semantics, queueing and worker scaling, batching and accelerator utilization, rate limiting, observability, and error handling.

  AnthropicSoftware EngineerML System DesignOnsite
• Design a high-concurrency LLM inference serviceHard

  This problem evaluates your ability to design a scalable LLM inference serving platform, focusing on throughput optimization, memory constraints, and scheduling strategies for long-running inference requests.

  AnthropicSoftware EngineerML System DesignOnsite
• Design a low-latency ML inference APIHard

  This question evaluates competency in ML system design for real-time, low-latency inference APIs, including multitenancy, SLO/SLI definition, feature retrieval, model serving and rollout strategies, observability, cost control, and security/compliance within the ML System Design category.

  AnthropicSoftware EngineerML System DesignOnsite
• Design a model downloaderMedium

  This question assesses the architectural design of a distributed, secure, and resilient system capable of deploying large-scale machine learning models to heterogeneous edge devices.

  AnthropicMachine Learning EngineerML System DesignOnsite
• Design a prompt processing backendHard

  Design a prompt processing backend evaluates ML product requirements, data/labeling, modeling, serving architecture, evaluation, monitoring, and trade-offs in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

  AnthropicSoftware EngineerML System DesignOnsite
• Design an LLM-based binary classifierMedium

  This question evaluates knowledge of LLM-based binary classification, probabilistic scoring and numerical stability (including log-prob handling and log-sum-exp), prompt construction, batching and sequence-level aggregation, calibration and evaluation metrics, plus analysis of failure modes and inference-time complexity.

  AnthropicSoftware EngineerML System DesignTechnical Screen
• Design an inference routing and scheduling layerHard

  This question evaluates system-design and production machine learning engineering skills, specifically distributed systems, routing and scheduling, multi-tenant isolation, caching, dynamic batching, and capacity planning for heterogeneous GPU/CPU inference backends.

  AnthropicMachine Learning EngineerML System DesignOnsite
• Estimate FFN Compute, Memory, and Sharding CommunicationHard

  Estimate standard FFN FLOPs, parameter and activation bytes, and sharding communication, then compare compute, HBM, and network limits with explicit layout assumptions.

  AnthropicSoftware EngineerML System DesignOnsite
• Estimate VRAM and compare model parallelismHard

  This question tests the ability to calculate memory overhead for transformer model parameters and activation buffers, while evaluating the architectural trade-offs between tensor and pipeline parallelism for large-scale training.

  AnthropicSoftware EngineerML System DesignOnsite
• Extend an LLM Inference Server with a Prefix KV CacheHard

  Anthropic Fellows OA practice: add exact-match, prefix and shared-prefix KV caching with an LRU token budget to an existing LLM inference server, keeping greedy outputs identical.

  AnthropicMachine Learning EngineerML System DesignOnline Assessment
• Performance-Model a Sharded Matrix Multiply: FLOPs, Memory, Network and RooflineMedium

  Model the performance of a matrix multiplication by hand: count floating-point operations, memory bytes and network bytes, apply the roofline model to decide what bounds it, and compare sharding along each dimension across several accelerators. It tests arithmetic intensity, collective communication costs and quantitative reasoning.

  AnthropicSoftware EngineerML System DesignOnsite
• Review an inference API design for scaleHard

  This question tests a candidate's ability to critically evaluate ML system designs for production-scale inference APIs, covering multi-tenancy, GPU resource constraints, and streaming token delivery. It assesses architectural reasoning across reliability, latency trade-offs, and capacity planning in the ML Systems domain.

  AnthropicSoftware EngineerML System DesignOnsite
• Review and Scale an Inference Service DesignMedium

  Tests the ability to architect a high-throughput ML inference service under strict latency SLAs, including dynamic batching, hardware capacity estimation, admission control, and multi-model routing.

  AnthropicSoftware EngineerML System DesignTechnical Screen
• Route and Batch Inference with Eight GPUsHard

  Design an inference routing, dynamic batching, and scheduling system across a fixed pool of eight 80GB GPUs serving both large and small language models under strict latency and memory constraints.

  AnthropicSoftware EngineerML System DesignTechnical Screen
• Schedule Large and Small Model Inference on an Eight-GPU PoolHard

  Design an inference scheduler for a shared 8-GPU node that co-locates single-GPU small models and all-reduce 8-GPU large models. The design must address gang scheduling, prefix cache reuse, deadline-aware priorities, and overload mitigation.

  AnthropicMachine Learning EngineerML System DesignTechnical Screen
• Choose File Storage Tiers Using Access PredictionsMedium

  Design an end-to-end machine learning and decision system that predicts file access likelihood to automatically transition cloud-stored user files across storage tiers to minimize operational cost while meeting retrieval latency SLAs.

  AppleMachine Learning EngineerML System DesignTechnical Screen
• Design App Store searchMedium

  This question evaluates a candidate's ability to design scalable, low-latency app marketplace search systems, covering competencies in query understanding, metadata ingestion and indexing, candidate retrieval and ranking, personalization, multilingual and typo-tolerant handling, cold-start and spam prevention, as well as online experimentation and monitoring. Commonly asked in the ML system design/search engineering domain to probe architectural trade-offs, measurement of product requirements and success metrics, and operationalization, it spans both high-level conceptual reasoning and practical application details.

  AppleMachine Learning EngineerML System DesignOnsite
• Design a 200-Terabyte Media Migration and Training Input PipelineHard

  This problem tests your ability to design a high-throughput data ingestion pipeline, focusing on network bottlenecks, storage strategies, and synchronization between on-premises assets and cloud-based training clusters.

  AppleSoftware EngineerML System DesignTechnical Screen
• Design a CPA system for ad biddingMedium

  This question tests the ability to design an end-to-end Cost-Per-Acquisition (CPA) bidding system, specifically focusing on how to model conversion probabilities and handle the complexities of delayed feedback and bid pacing.

  AppleMachine Learning EngineerML System DesignTechnical Screen
• Design a grounded voice assistantMedium

  This problem evaluates your ability to design a retrieval-augmented generation (RAG) system for a voice assistant, focusing on grounding model outputs in trusted sources to prevent hallucinations.

  AppleMachine Learning EngineerML System DesignOnsite
• Design a streaming embedding-based classifierHard

  This question evaluates a candidate's ability to design end-to-end streaming machine learning systems, including online text preprocessing, tokenization, embedding generation, continuous model training, and low-latency classification serving.

  AppleMachine Learning EngineerML System DesignTechnical Screen
• Optimize image filters on deviceMedium

  This question evaluates competency in on-device machine learning and performance engineering, covering algorithm choice, numerical precision trade-offs, hardware utilization (CPU/GPU/accelerator), memory layout, tiling/streaming, and trade-offs among latency, memory overhead, and battery impact.

  AppleMachine Learning EngineerML System DesignOnsite
• Debug a GPT training pipelineMedium

  This question evaluates debugging and implementation skills for transformer-based training pipelines, specifically attention masking, training-loop correctness, positional encoding integration, and the ability to design unit tests that catch such bugs.

  Applied IntuitionMachine Learning EngineerML System DesignTechnical Screen
• Design Jira bug-to-team classification systemMedium

  This question tests the ability to design an end-to-end ML pipeline for classifying incoming Jira tickets to appropriate engineering teams based on textual and historical metadata.

  AtlassianMachine Learning EngineerML System DesignOnsite