Katkat
Back

Design a creator posting-frequency experiment

TikTokData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests the ability to design a controlled experiment while accounting for network interference, non-compliance, and selection bias in a two-sided platform.

Problem Statement

TikTok wants to determine if sending a weekly "Creator Nudge" (a push notification or email) increases the number of videos posted by creators. We suspect that creators who see more engagement from their followers are more likely to post, creating a potential spillover effect where the activity of a creator in the treatment group could influence the behavior of creators in the control group via shared audience interaction.

Design an experiment to evaluate the efficacy of the nudge. Your design should specifically address:

  1. Randomization Strategy: How will you define the unit of randomization and mitigate the risk of spillover effects caused by interconnected creators or shared viewers?
  2. Eligibility Criteria: How do you define the "target" creator population, and how do you handle creators who are inactive or rarely open push notifications?
  3. Metric Selection: Define the Primary Metric (the main indicator of success) and secondary Guardrail Metrics (to ensure the nudge does not negatively impact the ecosystem).
  4. Sample Ratio Mismatch (SRM): How will you perform sanity checks to ensure the experiment setup is valid?

Provide your proposed design in a structured approach, justifying your choices regarding the trade-offs between clean causal inference and platform-wide impact.

Hints

Hint 1

Consider clustering randomization by geographic region or interest-based creator cohorts rather than individual creator IDs to isolate the spillover effects.

Hint 2

Think about the difference between Intent-to-Treat (ITT) analysis and Treatment-on-the-Treated (TOT) analysis when dealing with creators who ignore the nudge.

Hint 3

Guardrail metrics should focus on negative sentiment, such as notification unsubscribe rates or a decrease in the quality of content (e.g., shorter videos or lower engagement per post).

Follow-up questions

  • If we observe a significant increase in posting frequency but a decrease in the average views-per-video, how would you interpret these results?
  • How would you adjust your power calculation if the treatment effect is expected to be heterogeneous (e.g., highly effective for mid-tier creators but ineffective for top-tier celebrities)?
  • How would you account for 'novelty effects' where the nudge works only for the first two weeks and then stops being effective?