What you'll learn
A random forest is an ensemble: many decision trees whose predictions are combined. The idea is the wisdom of crowds — a diverse group of decent guesses averages out to a better answer than any single expert. Random forests are a superb default for tabular data: accurate, robust, and hard to misuse.
- What bagging is and why averaging reduces variance
- The two sources of randomness that make the trees diverse
- How out-of-bag error gives you free validation
- How to read feature importances — and their caveats
The problem with a single tree
As covered in the decision trees guide, one tree has high variance: change a few training rows and you can get a completely different tree. It also overfits easily. Random forests attack both problems by building many trees and letting them vote (classification) or averaging their outputs (regression).
Ingredient 1: bagging (bootstrap aggregating)
Each tree is trained on a different bootstrap sample — a dataset the same size as the original but drawn with replacement, so some rows appear multiple times and others not at all. Because each tree sees slightly different data, they make different mistakes. Averaging many such trees cancels out the individual errors — this is why the ensemble's variance is far lower than any single tree's.
Ingredient 2: random feature selection
Bagging alone is not enough: if one feature is very strong, every tree splits on it first and the trees end up correlated. So at each split, a random forest considers only a random subset of features (typically √n features for classification). This forces variety, decorrelates the trees, and is the 'random' in random forest.
Out-of-bag (OOB) error: free validation
Because each bootstrap sample leaves out roughly a third of the rows, every tree has a built-in validation set: the rows it never saw. Averaging each tree's error on its own left-out rows gives the out-of-bag error — an unbiased performance estimate without a separate validation split. Enable it with oob_score=True.
Implementing it in Python
We classify wine cultivars and inspect which chemical features matter most:
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
X, y = load_wine(return_X_y=True)
rf = RandomForestClassifier(n_estimators=300, max_features="sqrt", random_state=42)
scores = cross_val_score(rf, X, y, cv=5)
print("cv accuracy:", round(scores.mean(), 4), "+/-", round(scores.std(), 4))
rf.fit(X, y)
import numpy as np
top = np.argsort(rf.feature_importances_)[::-1][:3]
print("top features:", [load_wine().feature_names[i] for i in top])Output:
cv accuracy: 0.9665 +/- 0.0207
top features: ['flavanoids', 'proline', 'color_intensity']With almost no tuning the forest reaches ~97% cross-validated accuracy. feature_importances_ ranks how much each feature reduced impurity across all trees.
Reading feature importance carefully
Importances are handy but have traps: they are biased toward high-cardinality features (many unique values), and correlated features split their importance between them. For a more trustworthy ranking use permutation importance, which measures how much accuracy drops when a feature's values are shuffled.
Tuning the forest
- n_estimators — more trees never hurt accuracy, only speed; a few hundred is a good start.
- max_features — the size of the random feature subset; the main lever on tree diversity.
- max_depth / min_samples_leaf — control how deep individual trees grow.
- n_jobs=-1 — trees are independent, so training parallelises across all CPU cores.
Random forest vs. gradient boosting
Both are tree ensembles, but they differ in philosophy. A random forest builds trees in parallel and averages them to reduce variance. Gradient boosting builds trees sequentially, each correcting the previous one's errors, to reduce bias. Boosting (XGBoost, LightGBM) often edges out random forests on accuracy but needs more careful tuning; random forests are the more forgiving default.
Key takeaways
- A random forest = many decorrelated decision trees, combined by voting/averaging.
- Diversity comes from bootstrap samples (bagging) plus random feature subsets at each split.
- It reduces the variance and overfitting of single trees, and gives free OOB validation.
- It is one of the strongest, lowest-effort baselines for tabular data.
Further reading: scikit-learn ensembles and the visual guide.