Design a Translation API That Batches Requests on a Fixed Clock

You are designing the backend for a hosted text-translation API. Clients submit one sentence at a time and wait for the translated result. To keep the translation model's GPU busy, the service does not run one sentence through the model at a time — it collects pending requests into a batch and runs the whole batch through the model together.

The interviewer gives you one fixed number to design around: every batch, regardless of its size, takes 80 ms to run on a worker. Your job is to design everything around the client, the batch formation policy, and the worker fleet under that assumption, and to be explicit about what the 80 ms does and does not include.

Constraints and Clarifications

• Treat the 80 ms batch-run time as a given constant for your core design.
• Peak request rate, the maximum batch size, the number of GPU workers, and the end-to-end latency target are not specified — state your own assumptions or ask for them.

Clarifying Questions

• Does the 80 ms cover only the model's forward pass, or does it also include moving the batch onto and off of the GPU?
• Is there a cap on how many sentences fit in one batch, and does the 80 ms hold at that cap?
• Is there one model deployment, or several language pairs that cannot share a batch?
• What request-per-second rate and what p99 latency target must the service hit?
• Are results delivered as a single response per sentence, or can a client subscribe to a stream?

What a Strong Answer Covers

• Capacity math. Derive how many sentences one worker can translate per second as a function of batch size, given the fixed 80 ms, and use that to size the worker fleet for a target request rate.
• Batch-closing policy. Decide what closes a batch — it filling up, a timer expiring, or a worker becoming free — and discuss the latency/utilization trade-off of each, especially under bursty or low-traffic load.
• Routing results back. A batch returns many translations at once; explain how each one reaches the right waiting client connection, and what happens if that client already disconnected.
• Failure handling. What happens to the rest of a batch if the model crashes partway, and how retries avoid re-running sentences that already finished.
• Scaling and admission control. How the fleet scales with load, and what the service does when it cannot keep up (queue, shed load, or degrade).

Part 1 — Core design

Design the path from an incoming request to a batch running on a worker and back to the client, under the fixed 80 ms assumption.

Part 2 — Pushing past the fixed assumption

The 80 ms assumption breaks once you account for GPU memory limits, variable sentence lengths, and multiple language pairs. Discuss how your design would need to change, and what you would measure in production to know if the assumption still holds.