Katkat
Back

Train vs Test Error Diagnosis, Overfitting Fixes by Model Type, and Cross-Validation

LyftMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

A machine learning fundamentals screen that asks you to diagnose models from pairs of training and test error, including the unusual case where the test error is the lower one. It continues with curbing overfitting in linear, tree-based and neural network models, and with setting up validation splits and k-fold cross-validation.

In a technical screen for a machine learning engineer role, the first part of the interview is a fundamentals discussion with three connected questions: what training and test error reveal about a model, how to fix overfitting in different kinds of models, and how to set up data splits for validation.

Clarifying Questions

  • What metric is the error, and what would a trivial baseline (predicting the mean or the majority class) score? Without a reference point, "high" and "low" are only relative.
  • Were the training and test sets drawn from the same population and time period, and how was the split made?
  • Is the training error computed with the final model in inference mode, or reported while training was running?
  • How large is the test set?

Part 1 — Reading training and test error

The interviewer shows five pairs of (training error, test error). They include these patterns:

  1. Low training error, high test error
  2. High training error, low test error
  3. Both low
  4. Both high

For each pattern, explain the likely causes and what you would check or change next.

What This Part Should Cover

  • Bias and variance read from the level of the training error and the size of the gap
  • Causes beyond model capacity: differences between the two sets, noise, leakage, measurement artifacts
  • Concrete checks that tell these causes apart
  • The next action for each pattern

Part 2 — Fixing overfitting by model family

How do you reduce overfitting in a linear model, a tree-based model, and a neural network? Give remedies specific to each family, not only generic ones.

What This Part Should Cover

  • Remedies for linear models
  • Remedies for single trees and for tree ensembles, which behave differently
  • Remedies for neural networks
  • How the strength of each remedy is chosen without touching the test set

Part 3 — Train, validation and test splits, and k-fold cross-validation

How do you divide data into training, validation and test sets, and how do you run k-fold cross-validation?

Clarifying Questions for this Part

  • Is the data ordered in time, and can the same entity (for example, the same user) appear in many rows?
  • Roughly how much data is there?

What This Part Should Cover

  • The role and rough size of each split
  • The k-fold procedure step by step, including which model is trained at the end
  • Split variants for imbalanced, grouped and time-ordered data
  • Leakage through preprocessing or duplicate rows

Hints

Hint 1

Level and gap: Look separately at how high the training error is and at how far apart the two errors are; each points to a different kind of problem.

Hint 2

When unseen data looks easier: If a model does better on data it never saw than on the data it was fit to, compare the two datasets themselves, not only the model.

Hint 3

Find the capacity knob: For each family, identify what lets the model fit noise, and the parameter or procedure that limits it.

Hint 4

One job per split: Decide what each set is allowed to influence, then check that nothing from the held-out data can reach training, including through preprocessing.

Follow-up questions

  • Your cross-validation score is consistently better than your test score. What could cause that?
  • When would you prefer a single validation split over k-fold cross-validation?
  • How do you tune hyperparameters and still report an unbiased estimate of performance?
  • With gradient-boosted trees, you stop training early based on the validation set. How does that change what the validation score means?