Katkat
Back

Route and Batch Inference with Eight GPUs

AnthropicSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenML System Design

Last updated

Design an inference routing, dynamic batching, and scheduling system across a fixed pool of eight 80GB GPUs serving both large and small language models under strict latency and memory constraints.

Scenario

You are tasked with designing the inference serving and scheduling layer for an API cluster with a single high-density node containing 8x 80GB GPUs connected via high-bandwidth NVLink. The system must concurrently serve two models:

  • Large Model (70B parameters, FP16/BF16): Requires multi-GPU tensor parallelism (TP) to fit into memory and meet generation latency targets.
  • Small Model (8B parameters, FP16/BF16): Fits comfortably on a single GPU with ample headroom for KV cache.

Your goal is to design the routing, batching, GPU allocation, and admission control architecture that maximizes throughput and GPU utilization while guaranteeing strict latency Service Level Objectives (SLOs) and preventing out-of-memory (OOM) failures during traffic spikes.

Requirements

Functional Requirements

  • GPU Allocation Strategy: Propose an allocation of the 8 GPUs between the Large and Small models to balance capacity based on expected workloads.
  • Routing & Scheduling: Route requests to the appropriate model instances. Implement continuous (iteration-level) batching and manage model-specific KV cache budgets.
  • Overload Handling: Define admission control, queuing policies, and degradation strategies (e.g., load shedding, backpressure, speculative routing) when arrival rates exceed capacity.

Non-Functional Requirements & Scale

  • Traffic Profile: Peak incoming load of 300 requests/sec (75% targeting the Small model, 25% targeting the Large model).
  • Prompt/Output Profile: Average prompt length is 1,000 tokens; average output length is 400 tokens.
  • SLOs: Time to First Token (TTFT) < 250 ms (P95); Inter-Token Latency (ITL) < 30 ms (P95).
  • Reliability: Zero GPU Out-of-Memory (OOM) crashes under arbitrary traffic bursts.

Discussion Expectations

  1. Hardware & Model Sizing: Calculate weight footprints and KV cache memory requirements per token. Justify your GPU partitioning (e.g., static split vs. dynamic reconfiguration, choice of TP degree).
  2. Serving Engine Architecture: Detail the request lifecycle from API gateway to router, engine queue, scheduler (prefill vs. decode scheduling), and execution on worker runtimes.
  3. KV Cache & Memory Management: Explain how memory is budgeted per GPU/worker and how block-based allocation (e.g., PagedAttention) prevents fragmentation.
  4. Overload & Failure Modes: Design an overload mechanism detailing queue eviction, priority tiers, client backpressure (HTTP 429 vs. retry-after), and prefill preemption.

Hints

Hint 1

Calculate the raw parameter footprint in BF16 (2 bytes per parameter) to determine the minimum tensor parallelism required for the 70B model before accounting for the KV cache.

Hint 2

Decoupling the prefill (compute-bound) and decode (memory-bandwidth-bound) phases or using chunked prefills can prevent long prompt processing from stalling active token generation.

Hint 3

To avoid OOMs completely, admission control must be token-budget aware, reserving KV cache blocks up to a hard ceiling before allowing a request into the active batch.

Follow-up questions

  • If the traffic shifts dynamically from 90% Small model to 90% Large model over the course of a day, how would you redesign the GPU allocation without causing significant serving downtime?
  • How would you integrate chunked prefill or disaggregated prefill/decode architectures within this 8-GPU constraint?
  • If clients specify a max_tokens parameter that is far higher than their actual generation length, how do you prevent severe KV cache underutilization caused by over-reservation?