Katkat
Back

Design a high-concurrency LLM inference service

AnthropicSoftware Engineer · Onsite
HardSoftware EngineerOnsiteML System Design

Last updated

This problem evaluates your ability to design a scalable LLM inference serving platform, focusing on throughput optimization, memory constraints, and scheduling strategies for long-running inference requests.

System Design: High-Concurrency LLM Inference Service

Design a serving platform capable of handling thousands of concurrent requests for a Large Language Model (e.g., a 70B parameter model). The system must support both low-latency token streaming and high-throughput batch processing.

Functional Requirements

  • Support streaming responses (server-sent events) to clients.
  • Provide an API to submit completion requests with configurable parameters (e.g., max tokens, temperature).
  • Support multiple model versions and atomic updates.

Non-Functional Requirements

  • Minimize Time To First Token (TTFT).
  • Maximize GPU utilization/throughput.
  • Handle sudden traffic spikes gracefully.

Constraints & Assumptions

  • Inference is GPU-bound; you have a fleet of nodes with high-VRAM GPUs.
  • KV Cache management is a primary bottleneck for memory.
  • The model is too large to fit on a single GPU and requires model parallelism (tensor/pipeline parallelism).

Design Deliverables

  1. High-level architecture: Explain how incoming requests are routed, queued, and processed.
  2. Scheduling & Batching: How would you implement Continuous Batching to handle requests of varying lengths?
  3. Memory Management: Describe a strategy to manage KV cache memory to avoid fragmentation and OOM errors.
  4. Scalability: How would you handle scaling the fleet during peak hours?

Hints

Hint 1

Consider how 'Continuous Batching' allows the system to insert new requests into the batch as soon as others finish, rather than waiting for the entire batch to complete.

Hint 2

Think about how 'PagedAttention' or similar virtual memory management techniques for KV cache can eliminate internal and external fragmentation.

Hint 3

Address the trade-off between throughput (larger batch sizes) and latency (waiting for more requests to fill a batch).

Follow-up questions

  • How would you implement request preemption if a high-priority request arrives while the GPUs are fully utilized?
  • If the model size exceeds the VRAM of a single node, how do you manage the communication overhead between nodes during inference?
  • How would you design a monitoring and observability pipeline to detect 'sluggish' nodes or specific token-length bottlenecks in real-time?