Katkat
Back

Estimate unbiased ad scores with many reviewers

GoogleData Scientist · Technical Screen
MediumData ScientistTechnical ScreenStatistics & Math

Last updated

This problem tests the ability to design an estimation framework that accounts for rater bias and variance, specifically applying concepts like hierarchical modeling or fixed-effect normalization to estimate latent variables.

The Problem

You are working on an ad-quality ranking system. You have a dataset of 10,000 ratings where 100 reviewers each scored 100 ads on a scale of 1-10. There are 1,000 total ads. Because reviewers are not calibrated—some are consistently harsh (lower mean) or have restricted ranges (low variance), while others are lenient—the raw average score for an ad is a biased estimator of its true quality.

Design a method to estimate the latent quality score for each ad. You must:

  1. Define a statistical model that corrects for rater-specific bias (mean) and scale-specific sensitivity (variance).
  2. Explain how you would account for the uncertainty of your estimates, especially for ads that received fewer or inconsistent ratings.
  3. Propose a final ranking metric that balances the estimated quality against the confidence in that estimate.

Example

  • Input: A table ratings with columns [ad_id, reviewer_id, score].
  • Goal: Provide an estimator Q(ad_id) such that Q represents a de-biased quality score.

Constraints

  • The matrix of ratings is sparse (each ad is rated by ~10 reviewers on average).
  • Assume a rater's harshness can be modeled as an additive bias, and their range/sensitivity can be modeled as a multiplicative scaling factor.
  • Your solution should avoid manual "re-weighting" of scores if a more rigorous probabilistic approach is available.

Hints

Hint 1

Consider modeling the score of ad i by reviewer j as Score_ij = alpha_i + beta_j + gamma_j * (Score_ij_observed), where alpha is the ad quality, and beta/gamma are rater parameters.

Hint 2

Think about using a Mixed Effects Model or Bayesian hierarchical approach to shrink individual ratings toward the global mean when sample sizes are small.

Hint 3

For the ranking metric, consider using the Lower Bound of the 95% Confidence Interval of your estimate to ensure you aren't over-ranking ads with high variance.

Follow-up questions

  • How would your approach change if you discovered that reviewers are "gaming" the system by giving 1s to competitors' ads and 10s to their own?
  • If we add a temporal component, how would you account for a rater whose harshness changes over time?
  • How would you evaluate the performance of your ranking model if you don't have ground truth labels for the actual quality of the ads?