Katkat
Back

Design a real-time model inference platform: deployment, scaling and monitoring

FaireMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

An ML system design question asking for a real-time model inference platform that teams use to serve trained models inside their applications' request paths. It tests safe model deployment and rollback, per-model autoscaling and batching, failure handling, and monitoring from infrastructure up to model quality.

Design a platform for real-time model inference: teams hand it trained models, and applications call it for predictions inside their own request path. The discussion covered how models are deployed onto the platform, how it scales, and how it is monitored.

Constraints and Clarifications

  • Not given: the number of models, request rates, latency targets, model types or hardware. Ask, or state your assumptions.

Clarifying Questions

  • How many models and teams will use the platform, and how different are the models (small tree ensembles versus large neural networks that need GPUs)?
  • What latency budget do callers have, and is the API synchronous request and response?
  • Where do model inputs come from: the caller's request, lookups in a feature store, or both?
  • What does traffic look like: steady, daily peaks, or sudden spikes?
  • How fast must a bad model version be rolled back, and who approves a promotion to full traffic?
  • Do some predictions combine several models, such as an ensemble or a cascade?

Hints

Hint 1

What a team hands over: Decide what a model version consists of besides its weights, and how a new version reaches production traffic without a single all-at-once switch.

Hint 2

Scale each model on the right signal: Models differ in size, hardware and traffic. Think about what limits the throughput of one replica, what signal should add replicas, and what happens while a new replica is still loading.

Hint 3

Healthy servers, wrong answers: A model can keep returning fast, valid responses while its predictions get worse. Decide what you can watch before the true outcomes are known.

Follow-up questions

  • A new model version looks fine offline, but a business metric drops after rollout. How should the platform have caught it, and how quickly can it roll back?
  • One model needs a GPU and receives bursty traffic. How do you meet its latency budget without paying for idle GPUs?
  • Hundreds of rarely called models must all be servable. How do you avoid one always-on replica per model?
  • Feature lookups take most of a request's latency. What can the platform do about it?