Katkat
Back

Investigate Falling Successful Orders in LA

DoorDashData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your ability to decompose a drop in a high-level business metric into granular segments, formulate testable hypotheses, and propose methodologies to identify the root cause.

Scenario

You are a Data Scientist at DoorDash. You notice that the 'Successful Order Rate' (defined as Completed Orders / Total Checkout Attempts) in the Los Angeles market has dropped by 8% week-over-week. Your manager asks you to investigate the root cause and propose a strategy to recover the metric.

Task

  1. Describe your systematic approach to isolating the issue. How do you distinguish between a technical bug, a marketplace imbalance, or an external factor?
  2. Propose three distinct hypotheses—one each for customers, dashers, and merchants—that could explain the drop.
  3. How would you determine if this change is statistically significant versus noise, and what data cuts would you prioritize during your exploratory data analysis?

Constraints

  • Assume you have access to granular event logs, historical marketplace data, and real-time app telemetry.
  • Focus on the causal framework; explain how you would validate each hypothesis using observational data or potential A/B test results.

Example

  • Input: A plot showing the 8% drop occurring suddenly on Wednesday at 6 PM.
  • Goal: Describe how you would investigate if this correlates with a specific app update, a sudden spike in surge pricing (Dasher supply shortage), or a localized weather event.

Hints

Hint 1

Start by partitioning the funnel to see if the drop occurs at the checkout button, the payment authorization step, or the courier assignment phase.

Hint 2

Consider the 'three-sided marketplace' dynamic: did a supply-side disruption (lack of dashers) lead to price increases that suppressed demand, or did a merchant-side issue reduce inventory availability?

Hint 3

Use a difference-in-differences approach by comparing LA to a similar 'control' market (e.g., San Francisco or San Diego) to account for macro-trends or seasonality.

Follow-up questions

  • If you find that the drop is driven by 'Dasher assignment latency,' how would you quantify the impact of this latency on future user retention?
  • Suppose your investigation reveals that the drop was caused by a buggy app update; how would you design an experiment to measure the recovery after the hotfix is rolled out?
  • How would you adjust your analysis if the drop in successful orders was concentrated only among a specific demographic of users or restaurant types?