Katkat
Back

Design Harmful Content and OOM Detection

DatabricksMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

This question tests the ability to design an end-to-end ML system for real-time safety, covering taxonomy, data strategies, model architecture, and operational monitoring.

Design a Harmful Content and Out-of-Memory (OOM) Detection System

Design a system for a large-scale data processing platform that detects two distinct failure modes in user-submitted workloads:

  1. Harmful/Malicious Content: Detection of toxic, illegal, or prohibited content within submitted text or code snippets processed by your platform.
  2. OOM Predisposition: Detection of workloads likely to cause Out-of-Memory (OOM) crashes before they consume excessive cluster resources.

Functional Requirements

  • Provide low-latency inference for real-time screening.
  • Support model retraining pipelines to adapt to evolving malicious patterns.
  • Enable automated mitigation (e.g., rejecting jobs, alerting users, or throttling resources).

Non-Functional Requirements

  • High throughput to handle thousands of concurrent job submissions.
  • High precision to avoid false positives (e.g., blocking legitimate but complex code).
  • Resilience against adversarial attempts to bypass the safety filter.

What your answer should cover

  • Taxonomy/Data: How do you define "harmful" and what data sources do you use to train an OOM predictor?
  • Model Strategy: What architectures would you use for content classification (NLP) vs. resource forecasting (time-series/tabular)?
  • System Architecture: Describe the pipeline from job submission to inference and the integration point in the execution flow.
  • Monitoring: How do you detect model drift, adversarial attacks, or performance degradation in production?

Hints

Hint 1

Consider using an embedding-based approach for semantic harmful content detection and a regression model on historical resource usage metadata for OOM prediction.

Hint 2

Think about the trade-off between intercepting jobs synchronously during submission versus asynchronously during execution to maintain low latency.

Hint 3

Address the feedback loop: how will you handle human-in-the-loop review for flagged content and re-ingest those labels to improve the model?

Follow-up questions

  • How would you handle an adversarial user who tries to split a harmful prompt across multiple small, seemingly innocuous jobs?
  • If your OOM predictor is too conservative, it wastes capacity; if it is too lax, nodes crash. How do you balance this via your loss function or thresholding?
  • What specific metrics would you monitor to decide when it is time to retrain the safety classifier?