Katkat
Back

Diagnose a Production Classifier and Defend Its Systems Design

SpacexMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This problem tests the ability to diagnose training-serving skew in machine learning models and the architectural considerations required to maintain consistency between offline evaluation and online production environments.

Scenario

You are working on a binary classifier at SpaceX designed to predict whether a specific sensor telemetry packet indicates a 'Critical System Anomaly' that requires immediate ground-team notification. Your offline metrics (AUC-ROC: 0.94) are excellent, but since deploying the model to production, the recall has dropped significantly. You have been asked to lead the post-mortem and design a more robust deployment pipeline.

Task

  1. Diagnostic Strategy: List three specific hypotheses to explain the discrepancy between your high offline performance and poor production recall. For each hypothesis, describe one specific data-gathering or observability step you would take to confirm or rule it out.

  2. System Design: Describe an architectural improvement to your current pipeline that minimizes the risk of training-serving skew. Consider how your model ingests feature data (e.g., streaming vs. batch) and how feature transformation code is shared between the two environments.

Example Context

  • Offline training data: Collected via historical log dumps, processed using Pandas in a Jupyter environment.
  • Online inference: Features are fetched from a low-latency key-value store (e.g., Redis) and transformed via a C++ microservice.
  • Potential Issue: You notice the C++ microservice uses a different library for calculating rolling averages than the Python script used for training.

Hints

Hint 1

Consider the 'Training-Serving Skew' concept, specifically focusing on whether feature calculation logic is identical in both environments.

Hint 2

Think about 'Data Drift'—could the distribution of incoming telemetry data differ significantly from the historical data used to train the model?

Hint 3

Evaluate if the offline evaluation process accurately replicates the temporal ordering of events that the model experiences in production.

Follow-up questions

  • How would you implement a 'Shadow Deployment' or 'Canary' strategy to validate this model before it is given authority to alert ground teams?
  • If the issue is caused by latency in feature availability, how would you design a 'Feature Store' to ensure the model uses the most recent, yet consistent, state?
  • What monitoring metrics would you track in production to detect model degradation early without having immediate ground-truth labels for every prediction?