Katkat
Back

How to evaluate a new homepage feature

PayPalData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

A PayPal data science technical screen on designing an A/B test to evaluate a new homepage feature. It covers experiment design and randomization, primary/secondary/guardrail metric selection with tradeoffs, sample-size and power estimation, and pitfalls like novelty effects and interference. The solution emphasizes value-aligned metrics (payment completion over CTR) and fraud guardrails specific to a payments product.

Question

PayPal is planning to launch a new homepage feature (for example, a new CTA module, personalized content, or a redesigned layout) and wants to decide whether to ship it. Design an experiment to evaluate the feature.

Assume this is a consumer-facing homepage experience for logged-in users (justify explicitly if you change scope), and that you can instrument homepage events and downstream funnel events (login, add payment method, send money, checkout, etc.).

Your answer should cover:

  1. Objective and hypothesis — State the experiment objective and the null vs alternative hypothesis.
  2. Experiment design — Propose an A/B test design: unit of randomization (and why it is appropriate), eligibility, assignment consistency, duration considerations, and key validity checks.
  3. Metrics — Propose and clearly define (numerator/denominator, event definition, time window):
  • one primary (success) metric,
  • secondary / diagnostic funnel metrics,
  • guardrail metrics.
  1. Metric tradeoffs — Discuss the tradeoffs among candidate metrics. For example, the feature might increase homepage engagement but also slow the page, distract users from core payment flows, or change fraud risk.
  2. Sample size — Describe how you would determine the required sample size, including the inputs needed (baseline rate/variance, minimum detectable effect, significance level, power, traffic allocation, clustering, etc.).
  3. Increase power — List practical ways to increase statistical power if traffic is limited, without compromising validity, and explain the tradeoffs.
  4. Pitfalls — Call out common pitfalls such as novelty effects, seasonality, user heterogeneity (Simpson's paradox), logging errors, and interference across users or devices.
  5. Ship decision — Define a decision rule for whether to ship.