Katkat
Back

Measure impact of bot mitigation via experiment

MetaData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This question evaluates experiment design and causal inference skills for platform analytics, covering interference management, metric and guardrail definition, power and duration calculations, eligibility and exclusions, novelty/adaptation considerations, and diagnostics for bot‑mitigation interventions.

You will measure the impact of deploying a bot-mitigation system that hides or rate-limits suspected bots’ comments. Design an experiment covering:

  1. Randomization unit and interference: justify account-level vs cluster (graph/community) randomization to limit spillovers; propose a stepped-wedge ramp.

  2. Primary success metrics: human-visible comments per human DAU, human creator retention, report rates; define guardrails (new-user activation, moderation backlog, latency).

  3. Eligibility and exclusions: how to handle already-flagged accounts and high-risk geos; pre-exposure period and CUPED to reduce variance.

  4. Power: assuming baseline human-visible comments/DAU = 5.5 and MDE = −2% total comments but +1% human-visible comments, outline sample-size and duration calculations for a 14-day test with day-level clustering.

  5. Novelty and adaptation: plan for novelty decay, contamination from bot migration, and adversarial learning. 6) Fallback when A/B is infeasible: difference-in-differences with synthetic controls and high-frequency pre-trends checks. 7) Diagnostics if FP rises: define on-experiment holdouts and stratified analysis (new vs veteran users, creators vs consumers). Deliver an analysis plan and stop/go criteria.