Katkat
Back

Design weapon-selling ad detection from posts

MetaMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

This question tests the ability to design an end-to-end multimodal classification system for detecting policy-violating content (weapon sales) while balancing latency, precision, and robustness against adversarial evasion.

Problem Statement

Design a scalable ML system to detect and flag posts attempting to sell weapons on a social media platform. The system must process both text and images to determine if a post violates the policy prohibiting the sale of firearms and other regulated weaponry.

Functional Requirements

  • Classify incoming posts as 'Violating' or 'Allowed'.
  • Support high throughput (millions of posts per day) with sub-second latency for real-time flagging.
  • Minimize false positives to avoid removing legitimate user content (e.g., historical discussions or hobbyist posts).

Non-Functional Requirements

  • Resilience against adversarial obfuscation (e.g., using slang, misspellings, or cropped images).
  • Scalability to handle multimedia content.
  • Mechanism for human-in-the-loop review for ambiguous cases.

Scope of Response

  1. Policy & Labeling: Define the ground truth criteria and how you would curate/label the training data.
  2. Architecture: Describe the high-level pipeline (ingestion, feature extraction, model inference, and output).
  3. Multimodal Fusion: Explain how you would combine signals from text (LLMs/transformers) and images (CNNs/ViTs).
  4. Operationalization: Detail your strategy for handling adversarial attacks and monitoring system drift.

Hints

Hint 1

Consider using an ensemble approach where text and image models run in parallel, followed by a light fusion layer to make a final decision.

Hint 2

Think about how to use active learning or semisupervised labeling to handle the scarcity of explicit 'weapon-selling' examples compared to generic images of items.

Hint 3

Address adversarial tactics by incorporating data augmentation techniques like character-level noise for text and geometric transformations for images.

Follow-up questions

  • How would you handle a sudden spike in 'coded language' or slang terms designed to bypass text-only filters?
  • If you have a limited budget for human labeling, how do you select the most 'informative' samples for review?
  • What metrics would you prioritize if the business goal shifts from minimizing user friction to strictly enforcing safety policy?