Katkat
Back

Design a ChatGPT-like serving system

MicrosoftSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a scalable, low-latency LLM inference system, focusing on request handling, KV-cache optimization, and horizontal scaling strategies.

Design a ChatGPT-like Serving System

Design a backend service that supports real-time text completion (streaming responses) for a large language model. Assume the system needs to serve a high volume of concurrent users while keeping latency within acceptable thresholds.

Functional Requirements

  1. Accept user prompts and return streaming token responses.
  2. Maintain conversation history for multi-turn interactions.

Non-Functional Requirements

  1. Low Time-To-First-Token (TTFT) and high token generation throughput.
  2. High availability and fault tolerance.
  3. Efficient memory management for context windows.

Your Answer Should Cover

  • API Design: Define the interface for the client to initiate a stream.
  • Architecture: Describe the components (Load Balancer, API Gateway, Inference Engines, KV Cache management).
  • Scaling: How do you handle spikes in traffic? How do you manage GPU resource allocation?
  • Optimization: Discuss techniques like PagedAttention or Continuous Batching to mitigate the memory bottleneck of the KV-cache.
  • Data Consistency: How do you manage session state if a request is routed to a different node in a multi-turn chat?

Hints

Hint 1

Consider how the KV-cache grows linearly with sequence length and how traditional memory allocation might lead to fragmentation.

Hint 2

Think about how to batch multiple requests together even if they arrive at different times to maximize GPU utilization.

Hint 3

Address the trade-off between keeping state local to a GPU node versus using a distributed cache layer.

Follow-up questions

  • How would you handle a scenario where one user's prompt is significantly longer than others in the same batch?
  • What monitoring metrics would you implement to detect performance degradation specifically related to model inference speed?
  • How does your architecture change if you need to serve multiple different models with varying parameter sizes?