Katkat
Back

Design a pricing experiment with network effects

InstacartData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question evaluates experimental design and causal inference competencies for a two-sided marketplace pricing change, including handling network effects/interference, choice of unit of randomization and treatment markets, metric selection and diagnostics, power/MDE estimation, and operational risk assessment within the Analytics & Experimentation domain for a Data Scientist role. It is commonly asked to assess practical application of randomized and quasi-experimental methods across heterogeneous geographies and operational constraints, requiring both hands-on experimental design skills and conceptual understanding of bias, spillovers, and diagnostic metrics.

Scenario

You want to launch a new pricing model that incentivizes shoppers to place/pick up more orders during rush hours in a two-sided marketplace (supply and demand interact). You suspect network effects / interference: changing prices for some users may affect availability, ETAs, or acceptance rates for others.

Task

Design an experiment to evaluate the new pricing model.

Constraints

  • A standard user-level A/B test may be invalid due to spillovers (interference) across users.
  • The marketplace has heterogeneous geographies with different baselines.

Requirements

Your design should include:

  1. Unit of randomization and why (e.g., geo/market-level).
  2. How you will choose treatment/control markets (e.g., matched pairs / lookalikes).
  3. Primary metric (north star) and a set of diagnostic + guardrail metrics.
  4. How you handle bias/confounding (seasonality, pre-trends, market differences).
  5. Ramp plan, duration, and how you’ll estimate power/MDE at a high level.
  6. Risks: spillovers across nearby markets, partial compliance, concurrent changes.

Output

Provide a clear experimental plan and analysis approach (e.g., difference-in-differences).