Katkat
Back

Explain why CTR rises but CVR unchanged

AmazonApplied Scientist · Onsite
MediumApplied ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to diagnose Simpson's Paradox or composition shifts in funnel metrics when engagement increases without a corresponding lift in final conversion.

Scenario

You are working on an Amazon product page feature (e.g., a new recommendation carousel). You observe that the Click-Through Rate (CTR) for the page has increased significantly (+5%), but the Conversion Rate (CVR), defined as (Purchases / Total Page Views), has remained statistically flat.

Your manager is concerned that the increased CTR is attracting lower-intent users, diluting the overall conversion quality.

Task

  1. Propose three potential hypotheses to explain why CTR might increase while CVR remains stagnant. For each hypothesis, identify the specific metric or data segment you would investigate to validate it.
  2. Suppose you suspect the shift is due to a change in traffic composition (e.g., more mobile vs. desktop users). How would you use a stratified analysis (e.g., Cochran-Mantel-Haenszel test or subgroup analysis) to determine if the CVR is truly stagnant or if it is being masked by Simpson's Paradox?
  3. If your analysis confirms that the conversion rate per clicked item has actually decreased despite the higher volume of clicks, what product or business recommendations would you provide to improve the overall purchase funnel?

Example Case

Consider a scenario where the new feature increases clicks from 'window shoppers' who look at many products but rarely purchase, compared to your baseline group of 'intentional buyers' who click fewer items but have a higher propensity to buy. Explain how this composition shift impacts the aggregate metrics.

Hints

Hint 1

Consider whether the increase in CTR is driven by high-intent users or if the feature is just drawing more noise-level traffic.

Hint 2

Think about the mathematical definition: CVR = (Clicks / Page Views) * (Purchases / Clicks). If the first term increases but the product remains unchanged, how must the second term behave?

Hint 3

Use stratified analysis to check if the CVR is stable within distinct user segments before drawing conclusions about the global average.

Follow-up questions

  • How would you account for 'novelty effects' if this experiment only ran for three days?
  • If the increase in CTR was driven by a 'clickbait' UI element, how would you measure the long-term impact on customer lifetime value (LTV)?
  • What statistical tests would you use to determine if the observed increase in CTR is due to chance vs. a genuine behavioral shift?