Katkat
Back

Estimate ATE, ITT, and TOT from experiment

NetflixData Scientist · Onsite
EasyData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to distinguish and calculate Average Treatment Effect (ATE), Intent-to-Treat (ITT), and Treatment-on-the-Treated (TOT) in the presence of non-compliance.

Scenario

Netflix runs a promotion for a new subscription tier. They randomly assign 10,000 users to a 'Treatment' group (offered the discount) and 10,000 users to a 'Control' group (no offer).

Due to technical issues, not everyone in the Treatment group successfully claimed the offer, and some users in the Control group found a workaround to access the offer anyway. The observed data is as follows:

  • Treatment Group (N=10,000): 6,000 users actually used the offer (compliers/always-takers), 4,000 did not.
  • Control Group (N=10,000): 1,000 users found a workaround to use the offer, 9,000 did not.
  • Average Spend (Treatment Group): $50
  • Average Spend (Control Group): $45

Tasks

  1. Calculate the Intent-to-Treat (ITT) effect of the promotion.
  2. Define the Treatment-on-the-Treated (TOT) effect in this context.
  3. Explain why the simple difference in means between those who used the offer and those who did not is likely biased.
  4. How would you calculate the TOT using the 'Instrumental Variables' approach given this data?

Hints

Hint 1

ITT is simply the difference in outcomes between the randomized groups, regardless of whether they took the treatment.

Hint 2

The TOT estimate accounts for the 'compliance rate' by scaling the ITT by the difference in treatment take-up rates between the two groups.

Hint 3

Selection bias occurs because users who actively seek out the offer (the workaround group) are likely different from those who do not, violating the independence assumption.

Follow-up questions

  • What assumptions must hold for the IV estimator to provide a valid estimate of the Local Average Treatment Effect (LATE)?
  • How would you handle a situation where the 'Always-Takers' behavior is significantly correlated with the outcome variable?
  • If the experiment were a cluster-randomized trial (e.g., randomized by region instead of user), how would your analysis approach change?