Katkat
Back

Build harmful-content text classifier

MetaMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteMachine Learning

Last updated

This problem evaluates your ability to design a robust text classification system for identifying harmful content, emphasizing strategies for data imbalance, labeling ambiguity, and real-time inference.

Problem Statement

You are tasked with building a production-ready machine learning system to classify social media posts as either 'harmful' or 'safe'.

Requirements

  1. Data Strategy: The dataset is highly imbalanced (harmful content represents < 1% of total traffic) and features subjective labeling (disagreement among human annotators). Propose a strategy for data collection, cleaning, and handling labeling ambiguity.
  2. Modeling: Choose an appropriate architecture. Consider the trade-offs between a lightweight model (e.g., Logistic Regression with N-grams) and a transformer-based model (e.g., DistilBERT) in the context of a high-throughput production environment.
  3. Evaluation: Since missing harmful content (False Negatives) is more critical than flagging safe content incorrectly (False Positives), how will you select a classification threshold? Which metrics will you track beyond Accuracy?
  4. Deployment & Monitoring: Describe how you would handle inference latency constraints (e.g., < 50ms) and how you would monitor the model for 'concept drift' after deployment.

Assumptions

  • Daily traffic volume is approximately 100 million posts.
  • You have access to a pool of human annotators for active learning.
  • The system must prioritize low latency without significantly compromising precision.

Hints

Hint 1

Consider using techniques like focal loss or synthetic oversampling (SMOTE) to address the extreme class imbalance.

Hint 2

For labeling ambiguity, consider using inter-annotator agreement scores (e.g., Cohen’s Kappa) to filter or weigh your training labels.

Hint 3

Discuss the use of knowledge distillation or quantization to maintain high performance while meeting strict latency requirements.

Follow-up questions

  • How would you handle 'adversarial' attacks where users intentionally misspell words or use emojis to bypass the classifier?
  • If your model flags a post as harmful, what feedback loop would you implement to improve the system over time?
  • How does your strategy change if you need to detect 'nuanced' harm, such as hate speech that relies on specific cultural context?