Design experiments and diagnose metric changes

You are a Product/Data Scientist at a food-delivery marketplace (customers, dashers/couriers, merchants). Answer the following product analytics & experimentation prompts.

Scenario A — “Top Dasher” program change

The company is considering a change to the Top Dasher program (a set of incentives/benefits intended to improve dasher supply and delivery quality).

1. List pros/cons of launching or expanding such a program.
2. Define a metric framework for evaluation:

• Primary success metric(s)
• Diagnostic metrics
• Guardrails (e.g., cost, quality, fairness)

1. What is the randomization unit for an experiment (dasher vs market vs time vs geo), and why? Discuss trade-offs like interference/spillovers.

Scenario B — Your test metric is worse than control

In an A/B test, the primary metric is lower in treatment than control.

1. What are the first checks you do before concluding the change is harmful?
2. How do you decide whether to stop, continue, or iterate the experiment?
3. What follow-up analyses help you understand why it got worse?

Scenario C — Order cancellation rate is high

The order cancellation rate has increased substantially.

1. How do you diagnose the problem end-to-end (data + product)?
2. Which orgs/systems (merchant ops, courier ops, pricing, dispatch, support, payments, app reliability, etc.) are likely impacted?
3. Propose hypotheses for root causes and describe how you would test them (experiments or quasi-experiments).

Scenario D — Merchant promotions: self-serve vs auto setup

The company is deciding between:

• Self-serve promotions: merchants configure their own discounts/promotions, or
• Auto setup: the platform automatically recommends/sets promotions.

1. Compare pros/cons for merchants, customers, and the platform.
2. Propose an experiment plan (success metrics, guardrails, duration).
3. Choose a randomization unit and discuss trade-offs (contamination, fairness, heterogeneous effects).