Katkat
Back

Design an experiment for pricing page redesign

IntuitData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question evaluates a candidate's competency in experiment design, causal inference, impact sizing, metric framework development, and statistical analysis in the context of a pricing page redesign that adds an annual subscription option.

A product team is redesigning a pricing tier page. Historically the page only offered a monthly plan; the redesign adds an annual plan option.

You are the Data Scientist partner for the launch.

Questions

  1. Impact sizing: What are the key ways this change could impact the business (positive and negative), and how would you estimate the expected impact magnitude before running anything?
  2. Metrics: Propose a metric framework:
  • Primary success metric(s)
  • Diagnostic metrics (to understand why it moved)
  • Guardrail metrics (to prevent harm)
  1. Experiment design & ship decision: Describe how you would design the experiment and make a ship / no-ship decision, including:
  • Unit of randomization and key segments to monitor
  • Sample size / power approach (MDE) and duration considerations
  • How you would analyze results (e.g., confidence intervals) and handle pitfalls (SRM, novelty, repeated exposure)
  1. No experiment possible: If you cannot run an A/B test (policy, engineering constraints, or all-users launch), how would you measure impact as credibly as possible?