Katkat
Back

How would you measure billboard impact?

AirwallexData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests your ability to design a quasi-experimental framework for measuring the offline impact of an OOH (Out-of-Home) advertising campaign on digital conversion metrics.

Problem Statement

Airwallex is considering a large-scale billboard advertising campaign in three major cities (London, New York, and Sydney). Unlike digital ads, billboard impressions are difficult to track, and there is no direct 'click' data. You are tasked with measuring the incremental lift in new business account sign-ups attributable to these billboards.

Design an analytics framework to measure this impact. Your solution must address the following:

  1. Metric Definition: Define the primary success metric and the time window for attribution.
  2. Methodology: Since you cannot run a true A/B test (you cannot randomly assign individuals to see a billboard in a city), propose a quasi-experimental design. How would you establish a counterfactual (the 'what if the billboard didn't exist' scenario)?
  3. Data Sources: Identify the datasets required to support your analysis.
  4. Bias Mitigation: How will you account for potential confounding variables, such as existing seasonal trends or concurrent digital marketing campaigns?

Constraints

  • Assume you have access to historical sign-up data per city.
  • Assume billboard locations are fixed and known.
  • You do not have access to individual-level physical location tracking (GPS) for privacy reasons.

Hints

Hint 1

Consider using a synthetic control method to create a 'weighted' control group from cities where no billboards were placed.

Hint 2

Think about how to use geographic clustering or regional 'Geolift' testing to compare conversion rates in areas with high billboard density versus areas without.

Hint 3

Address the issue of time-lagged attribution; people rarely sign up for a business account the moment they see a billboard while driving or walking.

Follow-up questions

  • How would you account for the fact that billboard exposure is highly correlated with someone already being interested in financial services?
  • If we observe a spike in sign-ups, how do we distinguish between the billboard impact and a generic organic growth trend?
  • How would you conduct a power analysis to determine how many months of data are needed to detect a 5% lift in conversion with 80% confidence?