Katkat
Back

Design a Chat Assistant Inference Service: Streaming, Free-Tier Limits, GPU Scheduling

OpenAISoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Design the serving system for a chat assistant built on a large language model: stream replies token by token, rate-limit a free tier alongside paid users, schedule requests onto GPUs efficiently, and meet latency targets. It tests streaming protocols, batching, GPU memory limits, rate limiting and overload handling.

Design the serving system behind a ChatGPT-style chat assistant: users send messages in a conversation, and a large language model running on GPUs generates the reply token by token.

Focus on four areas: streaming the reply to the user as it is generated, rate limiting for a free tier alongside paid tiers, scheduling requests onto GPUs, and latency.

Clarifying Questions

  • What scale is expected: daily users, peak concurrent generations, typical prompt and reply lengths?
  • What latency targets apply, for the first token and for the tokens after it, and do they differ by tier?
  • How should the free tier be limited: by messages, by tokens, by time window, or by access to certain models?
  • Are there several models of different sizes, and can a request fall back to a smaller model under load?
  • How long is the conversation context, and is conversation history stored server-side?

Hints

Hint 1

Two latencies, not one: Separate the time until the first word appears from the pace at which the rest of the reply arrives, and ask which parts of the system control each.

Hint 2

The GPU's scarce resource: Think about what limits how many conversations one GPU server can generate for at once, and how the scheduler should admit, batch and evict work around that limit.

Follow-up questions

  • A user's connection drops halfway through a long reply. What happens to the generation, and what does the user see on reconnect?
  • How would you reuse computation for users who share the same long system prompt?
  • How would you roll out a new model version with no drop in quality or latency?
  • During a traffic spike, which users get degraded first, and how?