Katkat
Back

Estimate a Launch Impact with Difference in Differences

PinterestData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

Tests your ability to design and evaluate a regional quasi-experiment using Difference-in-Differences (DiD), validate key identifying assumptions, structure robust inference, and address modern complications such as staggered rollout and network spillovers.

Scenario

Pinterest recently launched a major visual commerce feature, "Shop the Look from Lens," aimed at driving save-to-purchase conversion. Because the feature required minimum local merchant density and coordinated creator marketing, a clean user-level A/B test was not feasible. Instead, the product team rolled out the feature across 20 designated market areas (DMAs) in the United States, while 80 other comparable DMAs remained on the existing experience.

You have daily, DMA-level aggregate metrics spanning 12 weeks before the launch and 8 weeks post-launch. Key metrics include weekly active Pinners (WAP), commercial clicks per active Pinner (clicks_per_user), and merchant gross merchandise value (GMV).

Your Task

Walk through an end-to-end quasi-experimental evaluation plan using Difference-in-Differences (DiD):

  1. Define the Estimand and Model Specification: State the primary causal estimand of interest. Write out the baseline two-way fixed effects (TWFE) regression equation at the DMA-week level. Define each variable, fixed effect, and index.
  2. Evaluating the Parallel Trends Assumption:
    • Detail how you would construct an event-study specification with dynamic leads and lags relative to the launch week t = -1.
    • Explain what patterns in the lead coefficients confirm or cast doubt on the identification strategy.
    • Describe two additional placebo or robustness checks you would run (e.g., in-time placebo, in-space/metric placebo).
  3. Inference & Uncertainty:
    • At what level should standard errors be clustered, and why?
    • Discuss the risk of standard asymptotic cluster-robust variance estimators when dealing with a moderate number of treated clusters (e.g., 20 treated DMAs), and propose an alternative inference method.
  4. Threats to Validity & Methodological Extensions:
    • Spillover effects: How could network-driven interactions (e.g., a user in a treated DMA sharing a product pin to a board followed by users in control DMAs) bias your DiD estimate? What direction would the bias take, and how would you detect or mitigate it?
    • Staggered Rollout: If the remaining 80 DMAs were treated in subsequent waves rather than left untreated, why does standard two-way fixed effects yield biased estimates (the "forbidden comparisons" problem), and what modern estimator would you use instead?

Hints

Hint 1

Formulate the TWFE regression using unit fixed effects for DMAs and time fixed effects for calendar weeks to absorb baseline regional differences and common temporal macroeconomic shocks.

Hint 2

For the event study, normalize the coefficient at the week immediately preceding the launch (`t = -1`) to zero, and verify whether pre-treatment coefficients are statistically indistinguishable from zero.

Hint 3

When adoption happens across multiple waves, standard OLS TWFE can assign negative weights to already-treated units used as controls; consider estimators like Callaway & Sant'Anna or Sun & Abraham.

Follow-up questions

  • If parallel pre-trends clearly fail in the raw data, how would you evaluate whether Synthetic Control or Synthetic Difference-in-Differences (SDID) is a suitable alternative?
  • With only 20 treated clusters, standard cluster-robust errors can over-reject the null. How does the Wild Cluster Bootstrap address this limitation?
  • How would you construct an 'exposure buffer' or network-distance metric to test whether control users connected to treated users show elevated baseline activity?