Katkat
Back

Diagnose and Validate a Regression Model

CitadelData Scientist · Technical Screen
EasyData ScientistTechnical ScreenStatistics & Math

Last updated

Work through regression model validation from split design and leakage prevention to residual analysis and segment-level error checks. The solution explains MAE versus RMSE, multicollinearity, coefficient limits, and when added model complexity is worth the operational cost.

You fit a regression model to predict a continuous outcome. The headline error metric looks acceptable, but the team asks whether the model is trustworthy. Describe how you would validate the model, diagnose its failures, and decide whether a more complex approach is warranted.

Constraints & Assumptions

  • The data may contain correlated features, outliers, missing values, repeated entities, or time ordering.
  • The prediction will be used on future observations from the same intended population.
  • Both predictive accuracy and explanation may matter.

Clarifying Questions to Ask

  • What is the unit of analysis, target, prediction horizon, and downstream decision?
  • Which errors matter most: absolute, squared, relative, or asymmetric errors?
  • Are observations grouped or time ordered, and which features are available at prediction time?
  • Does the team need causal interpretation of coefficients or only prediction?

Follow-up questions

  • What pattern in residuals would suggest a missing nonlinear relationship?
  • How would you handle a target with a long right tail?
  • Why can a coefficient change sign when another feature is added?
  • How would you monitor the regression after deployment?