Katkat
Back

Design an A/B test for search ranking

GoogleData Scientist · HR Screen
EasyData ScientistHR ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design a statistically sound A/B test for a search ranking algorithm, focusing on metric selection, hypothesis formulation, and common pitfalls in experimentation.

Scenario

The search team at a large tech company wants to launch a new ranking algorithm that prioritizes 'content freshness' in search results. You are tasked with designing an A/B test to determine if this change improves the user experience.

Your Task

Describe how you would design this experiment. Your response should address the following:

  1. Hypothesis: Formulate a clear null and alternative hypothesis.
  2. Metrics: Identify your primary metric (the one that determines success) and guardrail metrics (to monitor for negative impact).
  3. Unit of Randomization: Explain how you will assign users to groups and why you chose that level (e.g., user-level vs. query-level).
  4. Potential Pitfalls: Identify at least two challenges or biases you might face in this experiment (e.g., novelty effect, interference between groups, or seasonal trends) and how you would mitigate them.

Example Context

  • Product: Search results page (SERP).
  • Target Audience: Global active users.
  • Current Concern: The team is worried that while 'freshness' might increase clicks on new news, it could decrease long-term satisfaction if users find older, high-authority evergreen content harder to locate.

Hints

Hint 1

Consider metrics that balance short-term engagement, like Click-Through Rate (CTR), with long-term quality signals, such as search abandonment or session duration.

Hint 2

Think about the difference between randomization at the 'user' level versus the 'query' level and how that impacts consistent user experience.

Hint 3

When dealing with ranking changes, consider if the 'Novelty Effect' might lead to a temporary spike in clicks that fades over time.

Follow-up questions

  • How would you calculate the required sample size if the baseline CTR is 10% and you want to detect a 0.5% relative lift with 80% power?
  • If your test shows a significant increase in CTR but a significant increase in search abandonment, how do you decide whether to roll out the feature?
  • How would you account for 'network effects' or interference if a user's search behavior impacts the ranking of results for other users?