Katkat
Back

Detect bots using comment distribution patterns

MetaData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your ability to define metrics for identifying automated behavior and your statistical intuition for validating those metrics against human behavior.

Scenario

You are a Data Scientist at a large social media company. Your team has noticed an influx of low-quality comments on popular public posts. You suspect a subset of accounts are automated bots. You have access to the following event log table: comments_log.

Schema

  • comment_id: Unique identifier for the comment.
  • user_id: Unique identifier for the user account.
  • post_id: Unique identifier for the public post.
  • created_at: Timestamp of the comment.
  • content_length: Number of characters in the comment.
  • is_reported: Boolean flag indicating if the comment was flagged by other users.

Task

  1. Propose three distinct behavioral metrics to distinguish bots from human users based on this data.
  2. Explain the statistical approach you would use to set a 'threshold' for these metrics to classify an account as a bot.
  3. How would you validate that your proposed detection system is not disproportionately flagging legitimate high-frequency users (e.g., power users or celebrities)?

Hints

Hint 1

Consider calculating the time delta between consecutive comments from the same user to identify non-human posting cadences.

Hint 2

Look for patterns in content length consistency; bots often reuse identical or near-identical templates.

Hint 3

Think about using a holdout set or a manual labeling sample to calculate precision and recall of your chosen thresholds.

Follow-up questions

  • If the bots begin introducing randomized delays between comments to mimic human behavior, how would your feature set need to evolve?
  • How would you account for the fact that bot behavior might be seasonal or tied to specific 'trending' topics rather than individual user behavior?
  • Describe the trade-offs between a high-precision and high-recall approach when choosing to automatically shadow-ban detected accounts.