Katkat
Back

Design A/B Tests for Banner Ad and Group-Story Feature

SnapchatData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

Evaluates A/B test design for a banner ad and a group-story feature in a social app. Strong answers define monetization, engagement, safety, and retention metrics, handle accidental clicks for ads, account for network effects in group stories, and set launch criteria with guardrails.

Scenario

Product decision cases:

  1. whether to add a banner ad,
  2. whether to launch a group-story feature.
Question

What key metrics would you track to measure the impact of adding a banner ad versus not adding it? How would you design and analyze an A/B test to decide if the banner should be launched, including handling accidental clicks? For the group-story feature, which success metrics would you define and how would you set up the experiment (units, sample size, duration) to decide on launch?

Hints

Think CTR, dwell time, efficiency, retention, experiment design, guardrails, segmentation.