Katkat
Back

Compute probability an account is fake

MetaData Scientist · Technical Screen
EasyData ScientistTechnical ScreenStatistics & Math

Last updated

This question tests the ability to apply Bayes' Theorem to update the probability of a hypothesis (an account being fake) given new evidence (an automated flag).

Problem Statement

You are working on Meta's integrity team. You have an automated detection system that flags accounts as "suspicious." You are given the following metrics:

  • The base rate: 2% of all accounts on the platform are fake.
  • True Positive Rate (Sensitivity): If an account is fake, the system flags it with 95% probability.
  • False Positive Rate: If an account is legitimate, the system flags it with 5% probability.

Calculate the probability that an account is actually fake, given that the automated system has flagged it as suspicious. Provide your answer as a percentage rounded to two decimal places.

Example

If the base rate were 10%, the True Positive Rate were 90%, and the False Positive Rate were 10%, the calculation would involve the total probability of an account being flagged (P(Flagged) = P(Flagged | Fake) * P(Fake) + P(Flagged | Real) * P(Real)).

Edge Cases to Consider

  • How would the posterior probability change if the False Positive Rate dropped significantly?
  • How does the "base rate fallacy" affect the interpretation of this result?

Hints

Hint 1

Use Bayes' Theorem: P(Fake | Flagged) = [P(Flagged | Fake) * P(Fake)] / P(Flagged).

Hint 2

Calculate the total probability of an account being flagged by summing the probability of a true positive and a false positive.

Hint 3

P(Flagged) = (0.95 * 0.02) + (0.05 * 0.98).

Follow-up questions

  • If we increase the threshold of our model to reduce the false positive rate to 1%, how does that impact the precision of our flags?
  • How would you communicate the accuracy of this system to non-technical stakeholders who might assume a 95% sensitivity means a 95% chance that a flagged account is fake?
  • If the base rate of fake accounts increases (e.g., during a coordinated bot attack), how does that affect the Positive Predictive Value of your detector?