Design a prompt processing backend

HardSoftware Engineer Onsite ML System Design

Last updated October 27, 2026

Design a prompt processing backend evaluates ML product requirements, data/labeling, modeling, serving architecture, evaluation, monitoring, and trade-offs in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Design a background processing backend for large-language-model prompts. Clients submit prompts via an API and later poll or receive callbacks with results. Specify APIs, job queueing and prioritization, worker pools, model routing, prompt versioning, idempotency keys, retries and dead-letter queues, result storage, and observability. Address scaling, cost control, rate limiting, PII/security, and SLAs. Follow-up: support streaming partial outputs and cancellation.

Design a batch inference APIHard
Anthropic · ML System Design · Onsite
Design a GPU inference APIHard
Anthropic · ML System Design · Onsite
Design a high-concurrency LLM inference serviceHard
Anthropic · ML System Design · Onsite
Design a Low-Latency GPU Inference ServiceMedium
Anthropic · ML System Design · Technical Screen
Design a low-latency ML inference APIHard
Anthropic · ML System Design · Onsite
Design a model downloaderMedium
Anthropic · ML System Design · Onsite