Katkat
Back

Implement Linear Regression Gradient Descent

DatabricksMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteMachine Learning

Last updated

This question tests the ability to implement batch gradient descent for simple linear regression from scratch, requiring the derivation of gradients for weights and bias, and understanding how to optimize convergence.

Linear Regression Gradient Descent Implementation

Implement a function gradient_descent(X, y, learning_rate, iterations) to perform batch gradient descent for a linear regression model. Assume the model follows the form y_pred = w * X + b.

Requirements:

  1. Input X is a list/array of feature values, and y is a list/array of target values.
  2. Initialize w (weight) and b (bias) to zero.
  3. Use the Mean Squared Error (MSE) as the loss function: Loss = (1/n) * sum((y_pred - y)^2).
  4. Perform the update steps for w and b simultaneously for the specified number of iterations.
  5. Return the final w and b as a tuple.

Example:

# Input
X = [1, 2, 3]
y = [2, 4, 6]
learning_rate = 0.01
iterations = 1000

# Output
# Final w, b (e.g., approximately 1.99, 0.03)

Constraints:

  • Assume X and y are of equal length n.
  • n >= 1.
  • Learning rate is a positive float.
  • Iterations is a positive integer.

Edge Cases to Consider:

  • What happens if the learning rate is too large?
  • How does the scale of input features affect convergence speed?
  • How would the implementation change if we switched from Batch Gradient Descent to Stochastic Gradient Descent (SGD)?

Hints

Hint 1

The gradient of the MSE loss with respect to weight `w` is `(2/n) * sum(X * (y_pred - y))`.

Hint 2

The gradient of the MSE loss with respect to bias `b` is `(2/n) * sum(y_pred - y)`.

Hint 3

Remember to update `w` and `b` using the calculated gradients multiplied by the negative learning rate at every iteration.

Follow-up questions

  • How would you modify this code to include L2 regularization (Ridge regression)?
  • What are the trade-offs between choosing a very small versus a very large learning rate in this implementation?
  • If the input features have vastly different scales (e.g., one feature ranges from 0-1 and another from 0-1000), how would that impact your gradient descent and how would you fix it?