Katkat
Back

Evaluate AI-assisted ads creation feature

MetaData Scientist · Onsite
EasyData ScientistOnsiteAnalytics & Experimentation

Last updated

This question assesses the ability to design an A/B test for a new AI feature, focusing on metric definition, experiment design, and mitigating potential trade-offs between advertiser efficiency and ad quality.

Scenario

Meta is launching a new 'AI-Assisted Ad Creation' tool that uses generative models to suggest headlines, ad copy, and background images based on a URL provided by the advertiser. The engineering team claims this will reduce the time it takes for an advertiser to launch a campaign and increase the total number of active campaigns.

Your Task

You are the Data Scientist assigned to the launch. Design an experiment to evaluate the impact of this feature. Specifically, address the following:

  1. Metric Selection: Define your primary North Star metric, two secondary metrics, and at least one guardrail metric to ensure user experience (ad quality) is not compromised.
  2. Experimental Design: How would you randomize the rollout? What potential biases (e.g., novelty effect, cannibalization) might you encounter, and how would you control for them?
  3. Trade-off Analysis: Suppose you observe a significant increase in the number of ads created, but a decrease in the click-through rate (CTR) per ad. How would you determine if the feature is a net positive for the platform?

Requirements

  • Provide a clear rationale for your chosen guardrail metric.
  • Explain how you would isolate the effect of the AI suggestions from the advertiser's existing manual editing behavior.
  • Keep in mind that ad quality is a subjective metric; propose a way to quantify it for your analysis.

Hints

Hint 1

Consider metrics that span both advertiser-side efficiency and consumer-side engagement.

Hint 2

Think about how to distinguish between 'low-quality' ads and 'high-volume' ads when measuring CTR.

Hint 3

Discuss how long the experiment should run to account for advertisers adjusting their workflows.

Follow-up questions

  • How would you measure the long-term impact on advertiser retention rather than just immediate campaign creation volume?
  • If the AI tool performs significantly better for smaller businesses than large enterprises, how would you report these findings to stakeholders?
  • What steps would you take to ensure the AI-generated content does not violate safety policies or generate hallucinated claims?