Katkat
Back

Measure Billboard Campaign Impact: Design, Bias, Test Strategy

PinterestData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question assesses your ability to design a quasi-experimental framework for measuring the offline impact of an online brand campaign on user acquisition, accounting for external noise and selection bias.

Scenario

Pinterest is launching a large-scale physical billboard campaign in New York City (NYC) and San Francisco (SF) to increase new user sign-ups. You are tasked with measuring the incremental lift in new registrations attributable to these billboards over a 30-day period.

The Challenge

  1. Design Strategy: Propose an experimental or observational design to estimate the campaign's causal impact. Since billboards cannot be randomly assigned to individual users, how do you construct a suitable counterfactual?
  2. Bias Mitigation: Identify two major sources of selection bias or external confounding factors (e.g., seasonal marketing trends, local news events, or platform-wide updates) that could invalidate your results. How would you adjust for these?
  3. Metric Validation: Describe the SQL/statistical approach you would use to calculate the 'lift.' If you observe a 5% increase in sign-ups in NYC compared to a control group, what specific diagnostic checks would you perform to verify this isn't just noise?

Requirements

  • Provide a clear methodology for selecting a control group.
  • Define the primary metric and any secondary metrics (guardrail metrics) you would monitor.
  • Explain how you would account for the 'spillover effect' if users see a billboard but sign up via a mobile app download prompted by other digital ads.

Example

If you choose to use 'Synthetic Control' as your method, explain how you would weigh cities like Chicago or Seattle to approximate the pre-campaign behavior of NYC/SF.

Hints

Hint 1

Consider using a Geo-Experiment approach where you compare the target markets against a weighted combination of non-targeted cities.

Hint 2

Think about using a pre-period baseline to calculate a 'difference-in-differences' estimate to account for fixed differences between cities.

Hint 3

Address the interaction effect by suggesting a 'Hold-out' group or checking for lift in post-campaign conversion rates to ensure the users are high-quality/retained.

Follow-up questions

  • How would you adjust your model if you discovered a sudden spike in organic traffic in SF due to a tech conference occurring during the campaign?
  • If the billboard campaign is intended to improve long-term brand sentiment rather than just sign-ups, how would you change your primary success metric?
  • How do you determine the statistical power required to detect a 2% lift given the high variance in daily sign-up counts?