Katkat
Back

Resolve a Conflict Between Metrics and Customer Feedback

AmazonSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteAnalytics & Experimentation

Last updated

This question tests your ability to troubleshoot conflicting signals in A/B testing and product analytics, requiring a structured approach to reconciling quantitative data with qualitative sentiment.

The Scenario

You are a lead engineer on an e-commerce checkout team. Your team recently deployed a feature that reduces the number of clicks required to complete a purchase by pre-selecting default shipping and payment methods.

  • Quantitative Signal: The A/B test results are statistically significant, showing a 3% increase in conversion rate and no change in return rates.
  • Qualitative Signal: Customer support tickets and social media feedback show a spike in complaints, with users claiming the new process is "confusing," "deceptive," or "forcing them to make mistakes" with their shipping addresses.

Your Task

Describe how you would investigate this discrepancy. Walk through your diagnostic process to determine if the metrics are misleading or if the sentiment is unrepresentative. If you find that both signals are accurate, explain how you would weigh these findings to decide whether to roll back, iterate, or keep the feature.

Requirements

  1. Identify at least three potential data-level explanations for why metrics might look positive while sentiment is negative.
  2. Detail the specific segments or data cuts you would examine to validate your hypothesis.
  3. Propose a framework for balancing quantitative business outcomes against qualitative customer experience metrics.

Hints

Hint 1

Consider whether the 'conversion' metric is masking a high short-term post-purchase support burden that hasn't materialized in the data yet.

Hint 2

Examine if the improvement in conversion is driven by a specific power-user segment while the vocal complaints are originating from a subset of users with complex account configurations.

Hint 3

Think about 'selection bias' in qualitative feedback: are the people writing to support representative of the silent majority who might be benefiting from the change?

Follow-up questions

  • How would you instrument the checkout flow to capture 'intent to change' before a user completes a transaction, so you have better data next time?
  • If the conversion lift is purely driven by mobile users but the complaints are coming from desktop users, how would you change your deployment strategy?
  • What specific 'counter-metrics' would you add to your dashboard to ensure that ease-of-use is being tracked alongside raw conversion rates?