Katkat
Back

Prove high-quality pixels improve ad performance

MetaData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design a rigorous causal experiment to validate a product hypothesis while accounting for selection bias, confounding variables, and operational constraints.

The Scenario

The Ads Quality team at Meta is considering a feature that automatically downranks or flags ads with 'low-quality' pixel assets (e.g., blurry images or distorted video). The product team hypothesizes that enforcing a 'High-Quality Pixel' (HQP) standard will improve user engagement (CTR and conversion rate) and long-term ad ecosystem health.

The Task

You are asked to design a study to determine if ads with HQP assets perform significantly better than those without.

  1. Define your primary, secondary, and guardrail metrics. Why did you choose these?
  2. Design an experiment to measure the causal impact of HQP assets on ad performance. How would you address the selection bias where better-funded advertisers are more likely to produce high-quality assets?
  3. How would you detect if the HQP policy has unintended negative consequences on ad inventory or advertiser spend?
  4. Propose a SQL-based framework to compare performance segments (HQP vs. non-HQP) while controlling for advertiser-level spend.

Constraints & Considerations

  • Assume you have access to logs containing: ad_id, advertiser_id, pixel_quality_score (1-10 scale), impression_count, click_count, and conversion_count.
  • Consider the trade-off between strict quality enforcement and the risk of reducing total ad supply.
  • Address how you would validate that any observed lift is not simply due to 'better advertisers' choosing to use HQP.

Hints

Hint 1

Consider implementing a randomized control trial (RCT) at the advertiser or ad-set level rather than comparing existing ads, to mitigate selection bias.

Hint 2

Think about using propensity score matching or stratification to group advertisers by historical budget and industry before comparing pixel quality performance.

Hint 3

Define guardrail metrics like 'Total Ad Auctions Lost' or 'Advertiser Churn Rate' to ensure you aren't sacrificing platform health for short-term engagement gains.

Follow-up questions

  • If we implement this and see a lift in CTR but a drop in total impressions, how would you calculate the 'Net Global Utility' of the change?
  • How would you adjust your analysis if the 'pixel_quality_score' is known to be noisy or prone to misclassification by the computer vision model?
  • Suppose the impact on conversion rate is statistically significant but small, while the impact on ad supply is large. How would you justify the decision to the product team?