Katkat
Back

Design an unsafe content detection system

PinterestMachine Learning Engineer · Onsite
HardMachine Learning EngineerOnsiteML System Design

Last updated

This problem evaluates your ability to design a scalable, multimodal machine learning pipeline for detecting policy-violating content on a large-scale visual discovery platform.

System Design: Unsafe Content Detection

You are tasked with designing a system to identify and moderate unsafe user-generated content (e.g., hate speech, violence, or sexual content) on a visual discovery platform that processes millions of image and text uploads daily. The system must operate with high precision to avoid false positives and low latency to ensure safe user experiences.

Requirements

Functional Requirements:

  • Classify content into specific violation categories (e.g., 'Violence', 'Graphic', 'Hate Speech').
  • Support multimodal inputs: images, text captions, and metadata.
  • Incorporate a human-in-the-loop (HITL) review process for ambiguous cases.

Non-Functional Requirements:

  • Latency: Inference must complete within 200ms to allow for blocking or flagging content before it is widely distributed.
  • Scale: Handle high throughput during peak hours (e.g., 50k requests per second).
  • Reliability: The system must remain operational even if one model component (e.g., image classifier) fails.

Deliverables

  1. Data Strategy: How do you handle imbalanced training data where unsafe content is rare compared to safe content?
  2. Model Architecture: Describe how you integrate visual and textual signals. How do you handle multimodal fusion?
  3. Serving Strategy: Propose an architecture for real-time inference, including how you manage model versions and candidate selection for human review.
  4. Metrics: Define success metrics beyond simple accuracy. How do you measure the impact on the user community?

Hints

Hint 1

Consider using an ensemble approach where lightweight models act as a primary filter to reduce the load on heavier, more accurate models.

Hint 2

Think about how to use embeddings or attention mechanisms to align visual features from CNNs/ViTs with textual features from Transformers.

Hint 3

Focus on an asynchronous workflow for human-in-the-loop: prioritize ambiguous cases (where model confidence is near the decision threshold) for manual review.

Follow-up questions

  • How would you adapt this system to handle adversarial attacks where users intentionally add 'noise' to images to bypass detection?
  • If a new type of policy violation emerges (e.g., a specific harmful viral trend), how would you update the system without retraining the entire model stack?
  • How would you measure and mitigate potential demographic bias in your detection model?