Design an Offline Inference Platform with Hardware-Aware Routing
Gimlet LabsMember of Technical Staff, ML Systems · OnsiteLast updated
Design an offline large-model inference platform that routes prefill and decode across profiled hardware pools while balancing deadlines, fairness, batching, transfer cost, retries, and utilization.
Interview Prompt
Design an offline large-model inference platform that can place different phases of work on different hardware pools, including older accelerators suitable for decode. Focus on job ingestion, batching, model and data placement, routing, prefill/decode coordination, retries, capacity, fairness, and how the router uses measured hardware characteristics rather than static assumptions.
Constraints & Assumptions
- Jobs are offline but have explicit completion deadlines and tenant priorities.
- Hardware pools differ in memory, bandwidth, compute, and failure rate.
- Intermediate model state can be expensive to transfer between phases.
- One tenant cannot monopolize the fleet with very large batches.
Clarifying Questions to Ask
- Which model families, sequence lengths, and quantization formats are supported?
- Are prefill and decode always separated, optionally separated, or colocated by default?
- What throughput, deadline, cost, and quality objectives drive routing?
Follow-up questions
- When should the router colocate prefill and decode despite lower device utilization?
- How would you prevent stale performance profiles from causing a routing collapse?
- What happens when a hardware pool disappears mid-job?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.