Katkat
Back

Design and evaluate an ads ranking algorithm

MetaAnalytics Engineer · Onsite
EasyAnalytics EngineerOnsiteMachine Learning

Last updated

This problem tests your understanding of the end-to-end machine learning lifecycle for an ads ranking system, focusing on feature engineering, model selection, and the transition from offline metrics to online business impact.

Scenario

You are an Analytics Engineer at a social media company. You are tasked with designing an ad-ranking system that decides which ads to display in a user's feed. The goal is to maximize total revenue while maintaining a positive user experience.

Task

  1. Define the objective function for the ranking model. How do you balance revenue against user engagement (e.g., clicks, hides, or negative feedback)?
  2. Propose a set of features (both user-side and ad-side) that would be predictive of ad performance.
  3. Explain how you would evaluate the model performance offline before deploying it to production.
  4. Describe how you would monitor the model once it is live to detect performance degradation or bias.

Constraints

  • The system must return ranked ads in under 100ms.
  • Assume you have historical data on ad impressions, clicks, and conversion events.
  • Your answer should focus on the logic and trade-offs rather than specific code implementation.

Hints

Hint 1

Consider the Expected Value framework: P(Click) * P(Conversion | Click) * Bid Price.

Hint 2

Think about how to handle the 'cold start' problem for new ads or new users.

Hint 3

Use metrics like AUC-ROC or LogLoss for offline validation, and A/B testing for online business impact.

Follow-up questions

  • How would you handle a sudden change in user behavior, such as a major holiday or a global event?
  • How does your feature engineering strategy change if you move from a linear model to a deep learning-based architecture?
  • What strategies would you use to mitigate 'position bias,' where ads at the top of the feed receive more clicks simply because of their location?