Katkat
Back

Design an LLM Request Batcher When Every Batch Costs a Fixed Time

AnthropicSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Design the serving path for a code-completion model whose accelerators run requests in batches. The interviewer says one batch always takes a fixed 250 ms. Tests batching policy, capacity math, latency budgets, routing results back to callers, failure handling, and questioning the fixed-latency assumption.

Your company is launching an inline code-completion feature in its IDE plugin. Each keystroke pause sends one prompt (the code around the cursor), and the plugin waits for a single completion. The model runs on accelerator servers. Running one request at a time wastes most of the hardware, so requests must be grouped into batches.

To keep the first round simple, the interviewer gives you an assumption: any batch, up to the maximum batch size, finishes in 250 ms.

Functional requirements

  • Clients send a prompt and receive exactly one completion, or a clear error.
  • A client may cancel. The plugin cancels when the user keeps typing.
  • Support a priority tier: paid seats must see lower latency under load.

Non-functional requirements (state your own if you'd like different numbers)

  • Peak load: 40,000 requests/second worldwide.
  • Maximum batch size per model replica: 64.
  • Latency target: p99 under 600 ms from the client's point of view, including network time.
  • Availability: 99.9%. Losing one accelerator host should not cause visible errors.

What to deliver

  1. The request path, from client to model and back, with every component named.
  2. The batch-forming policy: what triggers a batch to run, and how that changes between low and high load.
  3. Capacity math: replicas needed at peak, with headroom, and the queueing delay that implies.
  4. How each result from a finished batch gets back to the connection that is waiting for it, and what happens if that client has already left.
  5. Failure handling: a replica crashes mid-batch, there is an overload spike, or a region is lost.
  6. Finally, challenge the assumption. What does the 250 ms actually include? What breaks when prompts and outputs have very different lengths, and how would you redesign for that?

Clarifying questions worth asking

  • Does the 250 ms cover prefill only, the full generation, or the whole round trip?
  • Does it hold for every batch size up to 64, and for every prompt length?
  • Is there one model, or several that can't share a batch?

Hints

Hint 1

With a fixed batch time, one replica's throughput is batch_size / 0.25 s. Compute it for a full batch and for a half-full one.

Hint 2

Run a batch when the replica is free and either the batch is full or the oldest request has waited for a set limit. At high load batches fill on their own; at low load the time limit caps latency.

Hint 3

Give each request an id and keep a map from id to the waiting connection on the front-end server that holds it. The batch worker answers that server, not the client.

Hint 4

Generation time grows with output length, so a fixed batch time doesn't hold for decoding. Look at continuous (iteration-level) batching.

Follow-up questions

  • Two model sizes share the fleet, and their requests can't share a batch. How do you divide the hardware between them as demand shifts?
  • How would you add streaming token output without increasing the number of open connections per gateway?
  • Prompts share long common prefixes (the same file header). How could prefix caching change your capacity math?
  • How do you load-test this before launch when real traffic patterns are unknown?