Katkat
Back

Design a weapon-sale ad detection system

MetaMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

This question tests the ability to architect an end-to-end multimodal classification pipeline for detecting policy-violating content at scale while balancing precision, recall, and operational latency.

System Design: Weapon-Sale Ad Detection

Meta is looking to build an automated detection system to identify and block advertisements or organic posts that promote the illegal sale of firearms or restricted weaponry. The system must process millions of daily uploads across text, image, and video modalities.

Requirements

Functional Requirements:

  • Detect content promoting weapon sales vs. benign content (e.g., historical discussion, news, or sporting goods).
  • Handle multimodal inputs: users often combine text captions with images of weapons.
  • Enable a fast-track reporting mechanism for community flags.

Non-Functional Requirements:

  • Scale: Handle high-throughput ingestion with sub-second inference latency for real-time blocking.
  • Accuracy: Minimize false positives (e.g., locking a user out for posting a photo of a prop sword) while maintaining high recall.
  • Adaptability: The system must quickly adapt to new slang or obfuscated image patterns used by bad actors.

Design Deliverables

  1. Data Strategy: How do you source, label, and manage data for a policy that is constantly evolving? How do you handle class imbalance given that illegal sales represent a tiny fraction of total content?
  2. Model Architecture: Describe the pipeline for multimodal feature fusion. How do you integrate text embeddings with image-based object detection?
  3. Deployment & Feedback: How do you handle edge cases (e.g., borderline content) and incorporate human-in-the-loop (HITL) workflows to improve model performance over time?
  4. Metrics: Define the success metrics beyond simple accuracy. How do you measure the impact on user experience vs. safety?

Hints

Hint 1

Consider using a two-stage architecture: a fast, lightweight classifier to filter out safe content and a more complex, expensive model for flagged items.

Hint 2

Think about how to utilize cross-modal attention mechanisms to correlate suspicious text keywords with specific visual objects in an image.

Hint 3

Address the feedback loop by explaining how human moderator decisions are fed back into the training set via active learning to combat distribution shift.

Follow-up questions

  • How would you handle adversarial attacks, such as users obfuscating weapon images with filters or using coded language in text?
  • If you have a limited budget for human labeling, what sampling strategy would you use to select the most informative examples for the model?
  • How would you design the system to respect regional variations in weapon policy laws (e.g., differences between US and EU regulations)?