Katkat
Back

Evaluate a New Ads-Ranking Algorithm

MetaData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to design a robust A/B test for an ads-ranking system while accounting for interference effects, budget constraints, and conflicting metric signals.

Scenario

Your team has developed a new ads-ranking algorithm intended to improve the relevance of advertisements shown to users. You are tasked with designing an A/B test to evaluate whether this model should be launched globally.

Your Task

  1. Experimental Design: Propose a randomization unit for this experiment. Discuss the pros and cons of using User ID versus Cookie ID or Request ID, specifically addressing how to handle network interference or advertiser budget leakage.
  2. Metric Strategy: The initial results show a 2% increase in Conversion Rate (CVR) but a 3% decrease in Click-Through Rate (CTR). Explain how you would interpret this divergence and what secondary metrics (e.g., Return on Ad Spend (ROAS), Total Spend, or Cost Per Acquisition (CPA)) you would analyze to determine the impact on advertiser value.
  3. Budget Interference: If the new algorithm surface ads that are more likely to convert but less likely to be clicked, advertisers might reach their daily budget caps faster or slower. How do you ensure the test accounts for the fact that advertisers may have limited budgets?
  4. Guardrails: Define three guardrail metrics to protect against degradation in user experience or long-term ecosystem health.

Constraints

  • Assume the platform supports millions of concurrent users and hundreds of thousands of active advertisers.
  • Provide a rationale for why your chosen randomization unit minimizes bias in the presence of budget-constrained advertisers.

Hints

Hint 1

Consider how a user-level randomization might lead to 'spillover' if an advertiser's budget is global across the platform.

Hint 2

Think about the funnel: if CTR drops but CVR rises, it suggests the algorithm is filtering out low-intent clicks and focusing on high-intent users.

Hint 3

Guardrail metrics should focus on negative user signals that might not be captured in immediate revenue calculations, such as ad fatigue or hidden/dismissed ad rates.

Follow-up questions

  • How would you adjust your analysis if you suspected a novelty effect in the first three days of the experiment?
  • If the experiment results show a significant increase in revenue but a decrease in user retention, how do you make the final launch decision?
  • How would you use a CUPED (Controlled-experiment Using Pre-Experiment Data) approach to increase the sensitivity of your primary conversion metric?