Katkat
Back

Design a Platform for Training and Serving ML Models

AmazonSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

This problem tests the ability to design an end-to-end MLOps platform capable of handling the full lifecycle of machine learning models, from experimentation to production deployment.

Problem Statement

Design an internal Machine Learning Platform (MLP) at a large-scale e-commerce company. The platform must support data scientists in training models, versioning them, and deploying them for both real-time online inference and large-scale batch processing.

Functional Requirements

  1. Experiment Tracking: Log training parameters, metrics, and dataset snapshots to ensure reproducibility.
  2. Model Registry: A centralized repository to version models (e.g., v1.0, v1.1-canary) and store model artifacts.
  3. Serving: Support low-latency online inference (e.g., p99 < 50ms) and high-throughput offline batch inference.
  4. Monitoring: Track drift in input data distributions and model prediction performance in production.

Non-Functional Requirements

  • The platform should support multiple ML frameworks (e.g., PyTorch, XGBoost, Scikit-learn).
  • High availability and scalability for serving infrastructure.
  • The system must handle frequent updates (model retraining) without significant downtime.

Deliverables

  • Provide a high-level architectural diagram (described in text).
  • Detail the API design for model registration and deployment.
  • Explain the data flow from training output to production serving.
  • Discuss the trade-offs between choosing container-based microservices versus serverless functions for serving.

Hints

Hint 1

Consider decoupling the training environment from the serving environment by using an intermediate object storage layer for model artifacts.

Hint 2

For online serving, think about how to handle different traffic routing strategies like A/B testing or canary deployments.

Hint 3

Address the challenge of 'training-serving skew' by defining a feature store or a shared preprocessing logic layer.

Follow-up questions

  • How would you handle a scenario where a model update causes a sudden increase in latency or a failure in upstream dependencies?
  • If your model needs to perform inference on huge datasets (petabytes), how would you design the batch processing pipeline to minimize costs?
  • How would you implement access controls and audit logging for sensitive datasets and model performance metrics across different teams?