Estimate an Experiment’s Mean Effect from Cohort and Metric Tables
LinkedInData Scientist · Onsite
Last updated
Estimate experimental impact from cohort and timestamped metric data using user-level outcomes, difference in means, confidence intervals, and valid testing.
Estimate an Experiment’s Mean Effect from Cohort and Metric Tables
An experiment cohort table records each user, experiment-entry time, and treatment or control assignment. A metric table contains timestamped values for each user. Explain how you would define an outcome, assemble the analytical dataset, and estimate the treatment effect as a difference in means with a confidence interval. Also explain how a hypothesis test relates to that interval. The metric definition is left to you and must be stated explicitly; this is a statistical design discussion, not a query-writing task.
Hints
Hint 1
Aggregate before comparing: A user with many metric rows should not automatically count as many independently randomized people.
Follow-up questions
- When does an absent metric row mean zero rather than missing data?
- What changes if the metric is a ratio rather than a per-user mean?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.