Katkat
Back

Evaluate a Product Change That Adds More User Options

MicrosoftData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to design an A/B test for a feature introducing choice-overload, specifically focusing on metric selection, trade-off analysis, and decision-making frameworks when metrics diverge.

Scenario

Microsoft is considering adding a "Custom Advanced Filters" menu to its cloud storage search interface. Currently, users have a single "Search" button that returns results based on broad relevance. The new feature allows users to filter by file type, modification date range, and owner.

The Problem

Design an experiment to evaluate this feature. Your response must address the following:

  1. Hypothesis: What is the core value proposition of adding these filters, and what is your hypothesis regarding user behavior?
  2. Experimental Design: How would you randomize users, and what is your primary metric of success?
  3. Metric Divergence: Suppose your experiment results show a 15% increase in "Advanced Filter" usage, but a 4% decrease in the overall file-download rate (the main goal of the search feature). How do you analyze this, and what is your framework for deciding whether to launch?

Constraints

  • Assume a large user base with high search volume.
  • Consider both short-term engagement and long-term user satisfaction.

Hints

Hint 1

Consider whether the increase in filter usage signifies higher user intent or added friction in the search journey.

Hint 2

Think about segmentation: does the drop in conversion happen for all users, or only for power users who might have been perfectly satisfied with the original broad search?

Hint 3

Use a framework like the HEART framework or GQM (Goal-Question-Metric) to connect low-level interaction metrics to high-level product goals.

Follow-up questions

  • How would you determine if the decrease in conversion is a result of 'analysis paralysis' or a sign that the search results are no longer relevant?
  • If you decide to launch the feature, how would you optimize the UI to mitigate the negative impact on the download conversion rate?
  • What counter-metrics or guardrails would you monitor to ensure the search quality hasn't degraded?