Compute diff-in-diff and pretrend flag
RobloxData Scientist · Online AssessmentLast updated
This question tests the ability to calculate a difference-in-differences (DiD) estimator for causal inference and implement a basic statistical test for pre-intervention trend divergence.
Scenario
You are analyzing a dataset from an observational study on a Roblox feature update. The dataset contains user_id, date, group (either 'control' or 'treatment'), and daily_sessions. The intervention occurred on 2023-10-01.
Task
- Calculate the Difference-in-Differences (DiD) estimate for the effect of the intervention on
daily_sessionsby comparing the treatment and control groups across the pre-period (2023-09-24to2023-09-30) and post-period (2023-10-01to2023-10-07). - Flag whether the pre-trend assumption is potentially violated. Define a violation as the difference in mean daily sessions between treatment and control groups changing by more than 10% between the first half of the pre-period (
2023-09-24to2023-09-26) and the second half of the pre-period (2023-09-27to2023-09-30).
Input Data
Assume the input is a list of dictionaries representing daily records:
data = [{'user_id': int, 'date': 'YYYY-MM-DD', 'group': str, 'daily_sessions': float}, ...]
Implementation
Write a function evaluate_experiment(data) that returns a dictionary:
{
'did_estimate': float,
'pretrend_violation': bool
}
Examples
- If
(Treatment_Post - Treatment_Pre) - (Control_Post - Control_Pre) = 5.0, thendid_estimateis 5.0. - Pretrend violation is
Trueifabs((T2 - C2) - (T1 - C1)) / abs(T1 - C1) > 0.1, whereT1/C1are means from the first pre-period block andT2/C2from the second.
Constraints
- Assume dates are provided in the specified range.
- Handle division by zero if
T1-C1is zero during the pre-trend check (returnFalseif denominator is zero).
Hints
Hint 1
Aggregate the data by group and time period (pre-vs-post) before calculating the difference of differences.
Hint 2
The DiD estimate formula is (Mean_Treatment_Post - Mean_Treatment_Pre) - (Mean_Control_Post - Mean_Control_Pre).
Hint 3
For the pretrend flag, partition the pre-intervention data into two distinct time windows and compare the group-mean differences.
Follow-up questions
- How would your approach change if the dataset contained high variance or outliers in daily sessions?
- What are the limitations of a simple pretrend check compared to a formal regression-based approach (e.g., event study)?
- How would you handle user churn or missing data points within the 'daily_sessions' metric?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.