Katkat
Back

Estimate bots and CI from DAU spike

MetaData Scientist · Onsite
MediumData ScientistOnsiteStatistics & Math

Last updated

This question tests the ability to decompose a mixture of legitimate user traffic and synthetic bot/CI traffic using statistical modeling, and requires justifying the choice of distribution for overdispersed count data.

Scenario

You are a Data Scientist at a large social media platform. You notice a 20% spike in Daily Active Users (DAU) over the last 48 hours. Your product team suspects this is caused by a new wave of automated bots or Continuous Integration (CI) test scripts rather than human organic growth.

Data Available

  • Hourly login count timestamps for the past 60 days.
  • User agent strings for all login events.
  • A categorical feature indicating if a user completed a 'human-only' interaction (e.g., CAPTCHA or high-latency UI navigation).

Tasks

  1. Propose a statistical framework to estimate the proportion of 'bot/CI' traffic vs. 'human' traffic during the spike. What distribution would you use to model the DAU counts, given that they are overdispersed (variance > mean)?
  2. How would you quantify the uncertainty of your estimate for the bot-to-human ratio?
  3. If the bot activity exhibits heavy-tailed behavior, why might a standard t-test be inappropriate for comparing the mean activity of suspected bots versus confirmed humans, and what alternative approach would you suggest?

Example Case

Suppose you model the daily logins Y as a mixture of two processes: Y = p * Bot + (1-p) * Human. If Human follows a Poisson distribution but the data shows significantly higher variance, suggest a transformation or a more robust distribution family to model this.

Hints

Hint 1

Consider using a Negative Binomial distribution to model the DAU counts, as it introduces an extra parameter to account for overdispersion compared to a simple Poisson model.

Hint 2

Expectation-Maximization (EM) algorithm or Bayesian latent class analysis can be used to estimate the mixture proportion 'p' when labels are missing or noisy.

Hint 3

For heavy-tailed distributions and uncertainty quantification, non-parametric bootstrapping or permutation tests are more robust than assuming normality.

Follow-up questions

  • How would you validate your bot detection model if you lack a 'ground truth' labeled dataset?
  • If the bots rotate their IP addresses and User Agent strings every hour, how would you adjust your feature engineering strategy?
  • How does the choice of prior distribution in a Bayesian framework impact your estimate when the signal-to-noise ratio is low?