Katkat
Back

Design a Model Inference API and Optimize Its GPU Usage

AnthropicSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

A system design interview question that asks you to design an inference API for a trained model served on GPUs, covering the public API, streamed responses, request queueing, scheduling onto GPU workers, overload handling and failure recovery. Follow-up questions probe how to raise GPU throughput and memory efficiency without missing latency targets.

Design an inference API: a service that lets clients send inputs to a trained machine-learning model running on GPUs and get the model's outputs back. The round covered the end-to-end design of the API and the serving system behind it, and the follow-up questions went into how to optimize the GPU side.

The model and the traffic were not specified. Treat them as clarifying questions; where you need a working assumption, a large language model that generates text token by token is a natural one, because it makes both the API and the GPU questions concrete.

Constraints and Clarifications

  • Not given: the model type and size, request volume, latency targets, input and output lengths, whether responses stream, and how many GPUs are available. Ask, or state your assumptions.

Clarifying Questions

  • What does the model do: generate text token by token, or produce one output per input in a single forward pass (classification, embeddings)?
  • How large is the model, and does it fit in one GPU's memory or must it be split across several GPUs?
  • Which latency matters most: time to the first output token, time to the full response, or throughput for offline batch jobs?
  • Do clients need streamed responses, and how long are typical inputs and outputs?
  • Who are the clients: is there per-customer authentication, quotas or rate limits, and are there priority tiers?
  • What request volume and burst pattern should the system handle?

Part 1 — The API and the serving system

Design the public API and the system behind it: how a request enters, where it waits, how it reaches a GPU, and how the result gets back to the client.

What This Part Should Cover

  • The API surface: request and response shape, streaming, errors, authentication, and limits
  • The request path from the gateway through queueing and scheduling to the model servers and back
  • How capacity is shared among clients: rate limits, admission control, and behavior under overload
  • Scaling and failure handling: adding model replicas, rolling out new model versions, and a GPU worker that dies mid-request

Part 2 — Getting more out of the GPUs

The interviewer then asked how to optimize the GPU side. Explain what limits throughput and latency on a GPU for this workload, and what you would change to serve more requests per GPU without missing the latency targets.

What This Part Should Cover

  • What bounds the time of a GPU step for this workload, and how grouping requests changes it
  • GPU memory management: model weights versus per-request state, and what limits how many requests fit at once
  • Techniques that trade precision, latency, or hardware count for throughput, each with its cost
  • How to measure the effect: throughput, latency, and GPU efficiency metrics

Hints

Hint 1

Where requests wait: GPUs are the scarce, expensive resource. Decide where a request waits when every GPU is busy, what decides which request runs next, and what the client sees while it waits.

Hint 2

Generation is not one step: If the model produces its output one token at a time, a request occupies a GPU for many steps, and different requests need very different numbers of steps. Think about what that does to the way you group requests and to how the response is returned.

Hint 3

Find what the GPU waits on: During one generation step for a single request, ask whether the GPU spends its time on arithmetic or on reading the model weights and stored per-request state from memory. Then ask what serving several requests in the same step changes about that.

Follow-up questions

  • A few clients send very long inputs and crowd out everyone else. How do you keep latency fair?
  • The model no longer fits in one GPU's memory. How do you split it, and what does that cost per request?
  • Traffic falls sharply at night. How do you avoid paying for idle GPUs without hurting the morning ramp-up?
  • A new model version must be rolled out without dropping in-flight streamed responses. How?