Katkat
Back

Explain variance reduction in random forests

LinkedInData Scientist · Technical Screen
MediumData ScientistTechnical ScreenMachine Learning

Last updated

This question tests your conceptual understanding of how Random Forests reduce the variance of base decision trees through bagging and feature randomness.

Variance Reduction in Random Forests

In the context of ensemble learning, Random Forests reduce the variance of individual decision trees without significantly increasing bias. Suppose you have a set of B identically distributed decision trees, where each tree T_i has variance sigma^2 and the pairwise correlation between any two distinct trees T_i and T_j is rho.

The Task

  1. Derive or state the formula for the variance of the average predictor T_avg = (1/B) * sum(T_i). How does this formula demonstrate the effect of the correlation rho on the ensemble's variance?
  2. Explain the specific mechanism through which Random Forests (as opposed to standard Bagging) attempt to decrease rho.
  3. Discuss the bias-variance trade-off: Why doesn't the continued addition of more trees (B -> infinity) lead to overfitting in a Random Forest?

Hints

Hint 1

Recall that the variance of the sum of correlated random variables involves the sum of individual variances plus the sum of all pairwise covariances.

Hint 2

Consider how the 'feature bagging' step (randomly selecting a subset of features at each split) influences the similarity between trees grown on the same bootstrap sample.

Hint 3

Remember that while the variance of the ensemble decreases as B increases, the bias of the ensemble is equal to the bias of a single tree, which does not change as you add more trees.

Follow-up questions

  • How does the choice of the hyperparameter 'mtry' (number of features to consider at each split) specifically impact the trade-off between individual tree strength and ensemble correlation?
  • If we had a dataset with very few features and one extremely dominant predictor, how would that affect the effectiveness of a Random Forest compared to a single decision tree?
  • How would the variance reduction properties change if the base learners were not trees but high-bias models like linear classifiers?