Katkat
Back

Estimate Portal’s causal lift on video-call usage

MetaData Scientist · Technical Screen
MediumData ScientistTechnical ScreenStatistics & Math

Last updated

This problem tests the ability to design an observational causal inference framework using Difference-in-Differences (DiD) in a setting with staggered adoption, specifically addressing selection bias and household-level measurement errors.

The Scenario

You are tasked with evaluating the impact of Portal (a hardware video-calling device) on user engagement. You have access to user-level daily call logs (minutes spent on video calls) and exact purchase dates for the device. Because the launch was not a randomized controlled trial, you must address the fact that device buyers are likely heavier video callers than non-buyers.

The Task

  1. Define the estimand (e.g., Average Treatment Effect on the Treated) and describe why a simple comparison of means between buyers and non-buyers would be biased.
  2. Propose a DiD model to estimate the effect of device adoption. Given that users purchase the device at different times, specify how you would structure the dataset to accommodate 'staggered adoption'.
  3. Address the following two complications:
    • Self-selection: Buyers already preferred video calling before the purchase.
    • Household spillover: Multiple people in the same household may use the device, but the purchase is linked to one user account.

Data Available

  • user_id: Unique identifier.
  • purchase_date: Date the device was registered to the account.
  • call_minutes: Daily aggregated video call duration.
  • household_id: Identifier grouping users living together.

Constraints & Considerations

  • Explain how you would validate the 'parallel trends' assumption.
  • Propose a strategy for handling household-level noise when the purchase is tied to a single user profile.

Hints

Hint 1

Consider the 'event study' approach, centering the time variable around each user's specific purchase date to align different adoption cohorts.

Hint 2

Use individual fixed effects to control for time-invariant unobserved differences between buyers and non-buyers, mitigating the selection bias.

Hint 3

To handle household spillovers, consider aggregating call minutes at the household level rather than the individual level to capture the full utility gain of the hardware.

Follow-up questions

  • How would you check for 'anticipatory effects' where users increase their call volume just before purchasing the device?
  • If you have access to a control group of users who browsed the product page but did not purchase, how would you refine your comparison group?
  • How does the interpretation of the 'lift' change if the device primarily shifts calls from mobile phones to Portal, rather than increasing total call time?