Katkat
Back

Slopes of y-on-x vs x-on-y Regression: Relationship and When They Are Equal

Two SigmaData Scientist · Technical Screen
HardData ScientistTechnical ScreenMachine Learning

Last updated

This question evaluates understanding of simple OLS linear regression estimators, moment-based expressions for slopes, the relationship between y-on-x and x-on-y fitted slopes, and how correlation and variable scaling influence coefficient values.

You fit two simple ordinary least squares (OLS) regressions, each with an intercept, on the same paired dataset $(x_i, y_i)$ for $i = 1, \dots, n$:

  1. Regress $y$ on $x$: $;y = \alpha_1 + \beta_{y|x}, x + \varepsilon$
  2. Regress $x$ on $y$: $;x = \alpha_2 + \beta_{x|y}, y + \eta$

The interviewer wants you to reason about how the two fitted slopes relate to each other, and what it takes for them to be equal.

Constraints & Assumptions

  • Both fits are plain OLS with an intercept and a single predictor.
  • The data are not degenerate: $n \ge 3$, and the sample variances of $x$ and $y$ are strictly positive.
  • "Coefficients" refers to the slope estimates $\hat\beta_{y|x}$ and $\hat\beta_{x|y}$.
  • You may transform the data (e.g., rescale variables) when asked how to make the slopes equal.

Clarifying Questions to Ask

  • Are both fits ordinary least squares, or is a symmetric method (e.g., orthogonal / total least squares) intended?
  • Should I treat the data as raw, or can I assume it has already been centered or standardized?
  • Do you care about the intercepts as well, or only the slopes?
  • When you ask how to make the coefficients equal, may I transform the variables, or must a condition hold on the raw data?

Part 1

Derive both slope estimates in terms of sample moments. What is the relationship between $\hat\beta_{y|x}$ and $\hat\beta_{x|y}$? In particular, what is their product, and why is $\hat\beta_{x|y}$ generally not equal to $1/\hat\beta_{y|x}$, even though the second regression looks like the first one "solved for $x$"?

Part 2

Under what condition on the data are the two slopes exactly equal? How would you transform the data so that the two regressions are guaranteed to produce the same slope, and what is the value of that common slope?

Hints

Hint 1

Write both slopes with sample moments: For simple OLS, the slope is the covariance over the variance of the predictor: $\hat\beta_{y|x} = \widehat{\mathrm{Cov}}(x,y)/\widehat{\mathrm{Var}}(x)$. Write the analogous expression for the reverse regression and compare the two.

Hint 2

Bring in the correlation: Express each slope as the sample correlation $r$ times a ratio of standard deviations. Multiplying the two expressions makes the relationship pop out — and connects it to a familiar goodness-of-fit quantity.

Hint 3

Set the two expressions equal: Using the moment forms from Part 1, equate them and see what must be true of the sample variances of $x$ and $y$ (watch for the degenerate zero-covariance case).

Hint 4

A standard preprocessing step does it: Think about what happens to both slopes if you z-score each variable. What do the standard-deviation ratios become, and what quantity is each slope then equal to?

Follow-up questions

  • Suppose both $x$ and $y$ are noisy measurements of the same underlying quantity. Is either OLS slope an unbiased estimate of the true relationship? What symmetric alternatives exist, and what do they assume?
  • In the $(x, y)$ plane, how do the two fitted *lines* (not just slopes) relate geometrically? When do they coincide?
  • What happens to each of the two slopes if you add independent noise to $x$ only, leaving $y$ unchanged?