Katkat
Back

Explain Core ML Concepts

InstacartMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question tests the conceptual understanding of supervised learning foundations, the mathematical relationship between regression techniques, and the convergence behavior of optimization algorithms.

Part 1: Linear vs. Logistic Regression

Explain the theoretical connection between Linear Regression and Logistic Regression. Specifically, describe how a Generalized Linear Model (GLM) framework allows us to transition from predicting a continuous variable to a probability. How does the choice of the link function and the assumed distribution of the target variable differ between these two models?

Part 2: Bias-Variance Tradeoff

Suppose you are training a model on a sparse, high-dimensional dataset common in e-commerce (e.g., user-item interaction matrices). If you significantly increase the capacity of your model (e.g., by adding high-degree polynomial features or increasing hidden layer depth), how do the bias and variance terms of the expected prediction error evolve? Provide a concrete example of a regularization technique that explicitly manages this tradeoff.

Part 3: Optimization Dynamics

Consider the objective function $J(w) = \frac{1}{2n} \sum_{i=1}^{n} (y_i - w^T x_i)^2$. Compare the convergence properties and computational requirements of Full-Batch Gradient Descent (BGD) versus Stochastic Gradient Descent (SGD) when training on a dataset with $n = 10^7$ records.

  • How does the variance of the gradient estimates differ between BGD and SGD?
  • Under what conditions might SGD converge to a better local minimum than BGD in non-convex settings?
  • Why is learning rate scheduling (e.g., decay) critical for SGD but less so for BGD when nearing the global optimum?

Hints

Hint 1

Think of the Bernoulli distribution for Logistic Regression versus the Gaussian distribution for Linear Regression.

Hint 2

Relate model complexity to the model's ability to fit training data versus its sensitivity to specific data points in the training set.

Hint 3

Consider the difference between a deterministic gradient calculated on the entire set versus an unbiased estimator based on a single sample.

Follow-up questions

  • How does the choice of loss function (MSE vs. Cross-Entropy) affect the sensitivity of your model to outliers?
  • If your model experiences high variance, would you prefer gathering more data or performing feature selection? Why?
  • How would the convergence behavior change if you used Mini-batch Gradient Descent instead of pure SGD?