Katkat
Back

Analyze time series and design validation experiment

GoogleData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests the ability to detect structural changes in time-series data and design a rigorous A/B testing strategy to measure the impact of an intervention on count-based metrics.

Scenario

You are a Data Scientist at a streaming platform. You notice a persistent, unexpected 15% drop in the daily count of 'New Subscriptions' starting approximately three weeks ago. The data is characterized by strong weekly seasonality (higher activity on weekends) and periodic spikes due to promotional marketing campaigns.

Task

  1. Change-Point Detection: How would you mathematically confirm that a structural break occurred at a specific timestamp, rather than observing random noise or seasonal variance?
  2. Counterfactual Forecasting: If you suspect a recent product update caused this drop, describe your approach to generating a counterfactual forecast to estimate what the subscription count would have been without the update.
  3. Experimental Validation: Design an experiment to validate whether a specific feature rollback will recover the lost subscriptions.

Constraints

  • The daily counts are non-negative integers.
  • You must account for the multiplicative nature of the weekly seasonality.
  • Your validation design must address the risk of 'spillover effects' if the treatment and control groups share common resources or social network clusters.

Hints

Hint 1

Consider using the CUSUM (Cumulative Sum) algorithm or Bayesian change-point detection to identify the inflection point while controlling for known seasonal patterns.

Hint 2

For forecasting, explore state-space models or Bayesian Structural Time Series (BSTS) which allow for the inclusion of external regressors like marketing spend and holiday indicators.

Hint 3

To address spillover effects in the experiment, evaluate the feasibility of cluster-based randomization or switchback testing rather than simple user-level randomization.

Follow-up questions

  • How would you handle missing data points if an upstream logging service was also failing during this period?
  • If the 'New Subscriptions' data shows significant overdispersion, how does that change your choice of model for the counterfactual forecast?
  • How would you communicate the confidence intervals of your estimated impact to stakeholders who expect a single definitive 'lost revenue' dollar amount?