Katkat
Back

Experiment Design to Measure Whether Voice Ordering Lifts Amazon Purchase Frequency

AmazonProduct Manager · Other
MediumProduct ManagerOtherAnalytics & Experimentation

Last updated

An Amazon product manager prompt asking how to design an A/B test for a new voice-shopping feature and learn whether it makes customers buy more often. It tests the choice of randomization unit, how purchase frequency is measured, sample sizing for noisy order counts, and guardrails such as accidental orders and order value.

You are a product manager at Amazon. A new feature that lets customers shop by voice is being tested. Design an A/B test that would show whether the feature makes customers purchase more often.

Clarifying Questions

  • Where does the feature run: on Alexa-enabled devices such as Echo speakers, as voice input inside the Amazon shopping app, or both?
  • Which customers can use it: everyone with a supported device, or only accounts with a payment method set up and voice purchasing turned on?
  • Does purchase frequency mean orders per customer across all of Amazon over a fixed window, or only orders in certain categories, such as reorders of household items?
  • What is the smallest increase that would justify investing further in the feature, and when is a decision needed?

Hints

Hint 1

Decide which orders count: A new way to buy creates orders through that channel almost by definition. Think about what the comparison must count so that the result shows extra buying rather than buying that simply moved from one channel to another.

Follow-up questions

  • Orders per customer rise in treatment, but average order value falls enough that revenue per customer is flat. What do you recommend?
  • Almost all of the lift comes from customers reordering the same household items. How does that change your view of the feature and the next test?
  • Leadership asks for a read after one week. What can you honestly say at that point, and what can't you?