Katkat
Back

Design a weapon-ad harmful content detection system

MetaMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenSystem Design

Last updated

This question assesses the ability to architect a scalable, multi-modal machine learning pipeline for detecting policy-violating content (weapon advertisements) within a high-throughput ad-delivery system.

Problem Statement

Design a backend system to detect and block advertisements promoting weapons or prohibited firearm accessories across a social media platform. The system must process millions of ad creatives daily, including images, videos, and text captions, before they are displayed to users.

Requirements

Functional Requirements

  • Real-time classification: Accept ad submissions and return a safety score/verdict.
  • Multi-modal analysis: Fuse signals from text (OCR and ad copy) and visual analysis (object detection in images/videos).
  • Human-in-the-loop: Route high-uncertainty samples to human reviewers.

Non-Functional Requirements

  • Low Latency: The classification must complete in <200ms to avoid impacting ad auction flow.
  • High Recall: Minimizing the amount of harmful content that reaches the platform is the priority.
  • Scalability: Ability to handle bursts in ad submissions during seasonal marketing events.

Your Answer Should Cover

  1. High-level architecture: The ingestion pipeline and service components.
  2. Model strategy: How you handle multi-modal inputs (e.g., late fusion vs. early fusion) and chosen model architectures.
  3. Data and Retraining: How you incorporate human labels and user reports back into the training loop.
  4. Trade-offs: Balancing precision (ad approval latency) versus recall (platform safety).

Hints

Hint 1

Consider using an ensemble approach where lightweight text classifiers filter obvious violations before invoking expensive video-processing models.

Hint 2

Address the challenge of 'adversarial content' by discussing how to handle obfuscated images or text-in-image attempts.

Hint 3

Think about the data flywheel: How do you specifically prioritize which samples to send to human reviewers to maximize model improvement?

Follow-up questions

  • How would you handle 'concept drift' if sellers start using new types of imagery or coded language to bypass your filters?
  • If your system experiences high latency, how would you implement a tiered inference strategy to prioritize high-risk ad categories?
  • How do you ensure the system remains fair and minimizes false positives for legitimate sellers, such as historical reenactment groups or movie prop businesses?