Katkat
Back

Review and Scale an Inference Service Design

AnthropicSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenML System Design

Last updated

Tests the ability to architect a high-throughput ML inference service under strict latency SLAs, including dynamic batching, hardware capacity estimation, admission control, and multi-model routing.

Scenario

You are tasked with designing and scaling an inference platform that serves multiple foundational ML models (e.g., embeddings, classification, and lightweight generative tasks) to internal and external clients. The system must meet strict tail-latency targets while maximizing hardware utilization across a fleet of GPU/accelerator instances.

Requirements

Functional Requirements

  • Provide a unified inference API supporting both synchronous evaluation and streaming responses.
  • Support dynamic request batching with a maximum batch accumulation window of 100 ms.
  • Host multiple model families with varying sizes and request arrival patterns across a shared or partitioned GPU cluster.
  • Implement admission control and overload protection to prevent cascading failures during traffic surges.

Non-Functional Requirements & Scale

  • Traffic: Peak load of 25,000 queries per second (QPS) globally across all models.
  • Latency: P95 latency under 150 ms (including network overhead and batch accumulation); P99 latency under 250 ms.
  • Availability: 99.99% uptime.
  • Cost Efficiency: Minimize idle GPU capacity via autoscaling and multi-model packing where feasible.

Key Areas to Cover

  1. Capacity Math & Hardware Sizing

    • Estimate the compute (TFLOPS) and memory bandwidth required for a representative model (e.g., 7B parameter FP16 model or 500M parameter embedding model) handling 5,000 QPS at batch size B.
    • Determine the required number of GPU nodes given VRAM footprints and memory bandwidth bottlenecks.

  2. Dynamic Batching & Scheduling

    • Detail the dynamic batching algorithm (e.g., continuous batching vs. naive time-window batching) and queue design.
    • Explain how the 100 ms batch window timeout interacts with arrival rates and request padding.

  3. Overload Control & Graceful Degradation

    • Design backpressure, rate-limiting, and queue eviction strategies (e.g., CoDel, priority drops, deadline-aware shedding).
    • Define mechanisms to signal degradation back to upstream services.

  4. Multi-Model Placement

    • Propose a placement and routing strategy: dedicated worker pools vs. multi-model co-location using unified memory or model offloading.
    • Address cold-start mitigation and model weight swapping overhead.

Hints

Hint 1

Differentiate whether your inference bottleneck is compute-bound (FLOPs) or memory-bandwidth bound (weights/KV transfer) to properly justify your capacity calculations.

Hint 2

For dynamic batching, consider continuous/iteration-level batching over standard static windowing to prevent short requests from blocking on long ones.

Hint 3

Implement load shedding using request deadlines rather than simple FIFO queues, dropping requests whose time-to-live will expire before GPU execution.

Follow-up questions

  • How does the introduction of variable-length prompt inputs affect your dynamic batching strategy and GPU memory fragmentation?
  • If one model suddenly experiences a 10x traffic spike, how do you prevent starvation of other co-located models on the same worker node?
  • How would you integrate speculative decoding or model distillation into this architecture to improve throughput without increasing GPU count?