Katkat
Back

How would you evaluate upranking shop ads?

MetaData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design an A/B testing framework for evaluating ad ranking changes, including metric selection, segment analysis, and identifying potential biases in historical data.

The Scenario

The product team proposes upranking Shop Ads (ads that lead to an in-app checkout experience) over Website Ads (ads that redirect users to an external advertiser URL). The hypothesis is that the frictionless in-app experience will increase conversion rates and user satisfaction, ultimately driving higher long-term value for the platform.

You are asked to design an experiment and evaluation strategy to validate this change. Specifically, address the following:

  1. Define primary and secondary metrics to measure the impact of this upranking.
  2. Identify critical guardrail metrics to ensure user experience or platform health is not degraded.
  3. Your initial results show that large advertisers see a net positive impact, while small advertisers see a net negative impact. How would you investigate these mixed results and what factors might explain them?
  4. Why might a historical comparison (comparing performance before vs. after the launch without an A/B test) lead to biased conclusions?

Your response should demonstrate a structured approach to experimental design, causal inference, and trade-off analysis.

Hints

Hint 1

Consider metrics across both the user side (e.g., clicks, time spent) and the advertiser side (e.g., ROAS, total revenue).

Hint 2

Think about negative externalities such as cannibalization of traffic or changes in ad inventory quality.

Hint 3

Consider how the platform's 'learning phase' for machine learning models or selection bias in which advertisers choose to use Shop Ads might influence results.

Follow-up questions

  • How would you handle the 'network effect' where an increase in Shop Ads might affect the auction price for Website Ads?
  • If the experiment results are statistically significant but the impact on total platform revenue is negligible, would you still recommend rolling out the feature?
  • How would you account for users who start their journey on a Shop Ad but finish their purchase on the advertiser's website?