Katkat
Back

Design an ads ranking ML system

SnapchatMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

This question tests the ability to design an end-to-end ads ranking pipeline, focusing on latency constraints, mitigation of position bias, and balancing multi-objective optimization.

Problem Statement

Design an ad-ranking system for a platform like Snapchat, where ephemeral content is interspersed with sponsored ads. The system must select and rank a subset of ads from a pool of thousands to display in a user's feed.

Requirements

  1. Functional: The system must return ranked ads in < 100ms. Consider the trade-off between retrieval (candidate generation) and ranking (scoring).
  2. Multi-Objective: The model should optimize for multiple goals: user engagement (CTR), advertiser value (eCPM), and platform health (preventing ad fatigue).
  3. Bias Mitigation: Ad performance is heavily influenced by its position in the feed. Design a strategy to handle position bias during training.
  4. Data Freshness: How will you handle real-time feedback (e.g., a user just clicked another ad) and ensure the system adapts to trending content?

Deliverables

  • Describe the high-level architecture (Retrieval -> Ranking -> Re-ranking/Filtering).
  • Propose a modeling approach for multi-task learning.
  • Explain your strategy for addressing position bias and delayed conversions (e.g., clicks happen now, but conversions happen hours later).
  • Define how you would measure success offline and online.

Hints

Hint 1

Consider using a two-stage architecture: a lightweight retrieval stage to reduce the search space and a deep neural network for precise scoring.

Hint 2

Address position bias by incorporating position as a feature during training but removing it or using a neutral default during inference.

Hint 3

For multi-objective optimization, explore Weighted Sum approaches or Multi-Task Learning architectures like Shared-Bottom or MMOE (Multi-gate Mixture-of-Experts).

Follow-up questions

  • How would you handle the 'cold start' problem for newly uploaded ads that have zero historical performance data?
  • If your model experiences a sudden distribution shift due to a holiday or viral event, what monitoring and recovery steps would you take?
  • How do you ensure no feature leakage occurs when joining real-time user activity logs with historical ad performance data?