Design a Double Descent Experiment

Scenario

Modern deep learning often violates classical learning theory: increasing model capacity or decreasing training sample size beyond the point of zero training error can actually improve test performance. This phenomenon is known as double descent.

You are asked to design a clean, reproducible synthetic experiment demonstrating sample-wise double descent using linear regression, and explain the underlying theoretical mechanics.

Experimental Specification

Describe the experimental setup to observe the risk peak as a function of sample size n relative to a fixed feature dimension d:

1. Data Generating Process (DGP): Define the distribution of features x, the ground-truth parameter vector w_star, and the target generation process y = x^T w_star + epsilon with additive noise variance sigma^2.
2. Model Formulation: Specify the estimator across both regimes:
   • Underparameterized regime (n > d): Standard Ordinary Least Squares (OLS).
   • Overparameterized regime (n < d): Minimum L2-norm interpolating estimator via the Moore-Penrose pseudoinverse.
3. Evaluation Protocol: Define the test risk calculation, how many independent trials per sample size n should be averaged to isolate variance from monte-carlo noise, and the specific range of n to evaluate around n = d.

Analytical Questions

In your proposal, provide direct technical answers to the following:

1. Matrix Conditioning: Explain mathematically why the expected generalization error peaks when n is approximately equal to d. How do the singular values of the empirical design matrix X behave as n approaches d?
2. Bias-Variance Decomposition: Break down the behavior of the bias and variance terms as n crosses the interpolation threshold from n d.
3. Effect of Noise: What happens to the double descent peak if the label noise sigma^2 is set to exactly 0? Does the peak still exist? Explain why or why not.