Katkat

Compute diff-in-diff and pretrend flag

RobloxData Scientist · Online Assessment

Last updated

This question tests the ability to calculate a difference-in-differences (DiD) estimator for causal inference and implement a basic statistical test for pre-intervention trend divergence.

Scenario

You are analyzing a dataset from an observational study on a Roblox feature update. The dataset contains user_id, date, group (either 'control' or 'treatment'), and daily_sessions. The intervention occurred on 2023-10-01.

Task

  1. Calculate the Difference-in-Differences (DiD) estimate for the effect of the intervention on daily_sessions by comparing the treatment and control groups across the pre-period (2023-09-24 to 2023-09-30) and post-period (2023-10-01 to 2023-10-07).
  2. Flag whether the pre-trend assumption is potentially violated. Define a violation as the difference in mean daily sessions between treatment and control groups changing by more than 10% between the first half of the pre-period (2023-09-24 to 2023-09-26) and the second half of the pre-period (2023-09-27 to 2023-09-30).

Input Data

Assume the input is a list of dictionaries representing daily records: data = [{'user_id': int, 'date': 'YYYY-MM-DD', 'group': str, 'daily_sessions': float}, ...]

Implementation

Write a function evaluate_experiment(data) that returns a dictionary:

{
    'did_estimate': float,
    'pretrend_violation': bool
}

Examples

  • If (Treatment_Post - Treatment_Pre) - (Control_Post - Control_Pre) = 5.0, then did_estimate is 5.0.
  • Pretrend violation is True if abs((T2 - C2) - (T1 - C1)) / abs(T1 - C1) > 0.1, where T1/C1 are means from the first pre-period block and T2/C2 from the second.

Constraints

  • Assume dates are provided in the specified range.
  • Handle division by zero if T1-C1 is zero during the pre-trend check (return False if denominator is zero).

Hints

Hint 1

Aggregate the data by group and time period (pre-vs-post) before calculating the difference of differences.

Hint 2

The DiD estimate formula is (Mean_Treatment_Post - Mean_Treatment_Pre) - (Mean_Control_Post - Mean_Control_Pre).

Hint 3

For the pretrend flag, partition the pre-intervention data into two distinct time windows and compare the group-mean differences.

Follow-up questions

  • How would your approach change if the dataset contained high variance or outliers in daily sessions?
  • What are the limitations of a simple pretrend check compared to a formal regression-based approach (e.g., event study)?
  • How would you handle user churn or missing data points within the 'daily_sessions' metric?