Katkat
Back

Design A/B Test for Cost-Per-Conversion Efficiency Analysis

TikTokData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

Design A/B Test for Cost-Per-Conversion Efficiency Analysis evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

You are the data scientist for a marketing division considering four new acquisition channels—YouTube ads, Google Search ads, Facebook ads, and direct-mail. Leadership wants the marketing budget used in the most cost-efficient way.

Question

How would you design an A/B (multi-arm) test to compare the cost-per-conversion efficiency of YouTube, Google Search, Facebook, and Direct Mail campaigns? What metric will you optimize and how will you define it precisely? State the null and alternative hypotheses and the statistical test you would apply. How will you determine sample size, budget split, and test duration given desired power and MDE? What post-hoc or follow-up analyses would you conduct after the main test?

Hints

Discuss cost-per-conversion metric, multi-arm design, power & alpha, ANOVA vs pairwise tests, budget allocation, assumptions, and demographic or creative differences.