Katkat
Back

Prove conversion ads value via incrementality

TikTokData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design a robust incrementality experiment to measure the causal impact of advertising, accounting for selection bias and interference.

The Scenario

You are a Data Scientist at TikTok working with a large retail brand. The client is running a conversion-focused ad campaign and claims their internal dashboard shows a 15% increase in total sales during the campaign period. However, you suspect this includes 'organic' conversions—users who would have purchased regardless of seeing the ads. The client wants to know the true incremental lift (the number of sales caused by the ads).

The Task

Design an experiment to measure the incremental conversion rate (iCR) of this campaign. Your response must address the following:

  1. Experiment Design: Propose a method to measure incrementality (e.g., Geo-Lift, Ghost Bidding, or PSA tests). Explain why you chose this method over standard attribution models.
  2. Metrics & Power: How would you define the primary metric? Given a baseline conversion rate of 2% and a target minimum detectable effect (MDE) of 0.2% absolute lift, describe how you would determine the necessary sample size.
  3. Causality & Interference: How will you account for potential cross-geo interference (e.g., users living in a test region but working in a control region)?

Constraints

  • The client has a fixed budget and is risk-averse regarding holding out ads from high-intent users.
  • Assume you have access to user-level identifiers, impressions, and purchase data.

Example

If you choose a Geo-Lift approach, specify how you would handle the 'selection' of test and control regions if the client only wants to launch the campaign in 50 out of 200 available metropolitan areas.

Hints

Hint 1

Consider the trade-offs between user-level randomization (e.g., PSA/Ghost bids) and cluster-level randomization (e.g., Geo-Lift) regarding potential network effects.

Hint 2

Think about how to use synthetic control methods to improve the precision of your experiment if your control group doesn't perfectly match your treatment group.

Hint 3

Address the 'carry-over' effect: how might the timing of the experiment influence the validity of the control group?

Follow-up questions

  • How would your experimental design change if you were measuring the impact on long-term brand equity rather than immediate conversion?
  • If the experiment results show a negative lift in the treatment group, what diagnostic steps would you take to determine if it's a data quality issue or a cannibalization effect?
  • How do you communicate the trade-off between experimental rigor and the lost revenue associated with holding out ads for the control group to a non-technical stakeholder?