Estimate OLS via streaming sufficient statistics
CitadelData Scientist · Technical Screen
Last updated
This question evaluates proficiency in streaming/out-of-core linear regression, including computing sufficient statistics with an intercept, assessing numerical stability of normal equations versus QR/SVD or incremental methods, incorporating ridge penalties, and designing parallel fault-tolerant computations.
You must estimate OLS coefficients β for very high-dimensional linear regression with data too large to fit in memory. (1) Derive how to compute XᵀX and Xᵀy in streaming mini-batches (include an intercept), then recover β and standard errors. (2) Discuss numerical stability vs. using QR or incremental/online methods. (3) Extend to ridge regression and show how to update with λI. (4) Explain how you would checkpoint and parallelize the computation.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.