Katkat
Back

Design metrics to detect harmful content and fraud

MetaData Scientist · Onsite
EasyData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests the ability to design robust monitoring metrics for detecting fraudulent user activity and policy-violating content, focusing on trade-offs between precision and recall.

Scenario

You are a Data Scientist at a social media company. Your team is concerned about an increase in 'coordinated inauthentic behavior' (CIB)—a type of fraud where fake accounts work together to artificially boost engagement on specific posts. The current system relies on basic threshold-based flagging, which is producing too many false positives and slowing down platform performance.

Task

  1. Define three primary metrics to monitor the effectiveness of a new detection system aimed at identifying CIB.
  2. Explain how you would differentiate between a spike in organic viral growth and a coordinated attack.
  3. Propose a method to validate whether a decrease in flagged content is due to better system performance or a failure in the detection logic.

Constraints

  • Assume you have access to logs of account creation timestamps, device IDs, IP addresses, clickstream patterns, and graph connectivity (follower/following).
  • The solution should account for the fact that fraudsters adapt their patterns quickly to avoid detection.

Example

If a post goes from 10 likes per hour to 1,000 likes per hour, how would you determine if this is a 'viral hit' versus 'bot activity'? Consider what specific data attributes (e.g., origin of traffic, user account age) you would compare.

Hints

Hint 1

Think about the trade-off between Precision (ensuring we only ban real fraudsters) and Recall (catching as many fraudsters as possible).

Hint 2

Consider graph-based features, such as the cluster density of users interacting with a specific post compared to the platform average.

Hint 3

Use a 'hold-out' group or periodic manual audit of flagged vs. non-flagged content to calibrate the system's accuracy over time.

Follow-up questions

  • How would you handle a scenario where fraudsters rotate their IP addresses and device fingerprints to mimic human behavior?
  • If your detection system has a latency of 500ms, how would you design a 'cascade' system that applies quick, low-cost filters before complex, high-latency models?
  • What statistical tests would you run to determine if an observed increase in fraud is statistically significant or just seasonal noise?