Katkat
Back

Implement and derive backprop from scratch

AnthropicSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteMachine Learning

Last updated

This question evaluates understanding and practical implementation of neural network fundamentals, specifically analytic backpropagation and gradient derivation, numerically stable binary cross-entropy computation, parameter initialization, gradient descent updates, and gradient checking.

From first principles, implement a tiny neural network for binary classification with one hidden layer. Write the forward pass, a cross-entropy loss, and the complete backward pass (no autograd). Derive analytic gradients for all parameters, implement gradient descent, and verify correctness with finite-difference gradient checking. Discuss numerical stability (e.g., log-sum-exp), initialization, activation choices, and how batch size affects variance.