Katkat
Back

Implement Simple Linear Regression with Gradient Descent from Scratch

DatabricksMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteMachine Learning

Last updated

Write simple linear regression trained with gradient descent from scratch, choosing the loss, learning rate and stopping rule yourself. Tests deriving the gradients correctly, writing a runnable training loop, understanding feature scaling and convergence, and verifying results against a closed-form fit.

Write simple linear regression trained with gradient descent, from scratch. Given training inputs x (one feature per example) and targets y, learn a slope w and an intercept b for the model $\hat{y} = w x + b$. Run gradient descent on a loss you choose and justify, and return the learned parameters. Do not call a library routine that fits the model for you.

The report records only the task itself. The loss, the stopping rule and the hyperparameters were not described, so choose them and explain your choices.

Clarifying Questions

  • Is this one feature only (simple regression), or should the code generalize to several features?
  • May I use NumPy for vector arithmetic, or must it be plain Python?
  • Should training be full-batch, stochastic or mini-batch?
  • How should the result be checked: against a closed-form solution, on held-out data, or with a unit test?

Hints

Hint 1

Start from the loss: Write the loss as an average over examples, then differentiate it with respect to each parameter before writing any code.

Hint 2

Watch the scale: Consider what happens with a fixed learning rate when the feature values are in the thousands rather than close to one.

Follow-up questions

  • How does the code change for many features, and when would you use the normal equations instead?
  • How would you add L2 regularization, and should the intercept be regularized?
  • What changes with stochastic or mini-batch updates, and how would you choose the batch size?