Katkat
Back

Design Harmful Content Detection

DatabricksMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

This problem tests the ability to design an end-to-end machine learning system for multimodal harmful content detection at scale, focusing on pipeline architecture, latency, and data governance.

Problem Statement

Design a scalable system to detect harmful content (e.g., hate speech, violence, or spam) in a feed of multimodal data, including images, short videos, and text captions, being uploaded to a platform at a rate of 100,000 requests per second.

Functional Requirements

  • Provide a real-time classification score for each piece of content.
  • Block or flag harmful content before it becomes visible to other users.

Non-Functional Requirements

  • Latency: Inference must complete within 200ms per request.
  • Scalability: The system must handle high throughput with burst capacity.
  • Robustness: High precision is required to minimize false positives for user-generated content.

Expectations

  • Define the high-level architecture (e.g., synchronous vs. asynchronous).
  • Explain your approach to handling multimodal data (fusion techniques).
  • Discuss the strategy for data labeling, handling edge cases, and continuous model improvement.
  • Outline how you would monitor the system for drift and performance degradation.

Hints

Hint 1

Consider using a multi-stage architecture where a lightweight filter screens content before passing suspicious items to more computationally expensive models.

Hint 2

Think about how to handle multimodal features: would you use separate encoders with a late fusion layer or an end-to-end transformer architecture?

Hint 3

Address the cold-start problem and how you would maintain a gold-standard dataset for iterative fine-tuning.

Follow-up questions

  • How would you handle adversarial attacks where users intentionally obfuscate harmful content (e.g., adding noise to images or using 'leetspeak')?
  • If the business requires a 99.9% recall threshold, how does this affect your infrastructure and labeling strategy?
  • How would you implement a human-in-the-loop system for ambiguous content that falls into the low-confidence classification range?