Decision Trees Explained: How Machines Learn to Ask the Right Questions (with Python)

What you'll learn

Decision trees are the most intuitive model in machine learning: they make predictions by asking a sequence of simple questions, exactly like a game of twenty questions. They handle numbers and categories, need almost no data preparation, and you can read the finished model like a flowchart.

• How a tree decides which question to ask at each step
• What Gini impurity and entropy actually measure
• Why trees overfit, and how depth limits and pruning stop it
• How to train and read a tree in scikit-learn

The intuition in plain English

Suppose you want to predict whether someone will enjoy a hike. You might first ask 'is it raining?' If yes, most people say no thanks — that question split your group cleanly. If no, you ask a follow-up: 'is the trail under 10 km?' Each question carves the data into groups that are more and more uniform in their answer.

A decision tree automates exactly this. It searches every feature and every possible cut-off to find the single question that best separates the classes, then repeats the process on each resulting group.

How a split is chosen: Gini and entropy

'Best separation' needs a number. Trees measure the impurity of a group — how mixed its labels are. A group that is all one class has impurity 0; a 50/50 mix is maximally impure. Two common measures:

Gini    = 1 - Σ (p_i)^2
Entropy = - Σ p_i * log2(p_i)

where p_i is the fraction of class i in the group. At each node the tree tries every candidate split and picks the one with the largest information gain — the biggest drop in impurity from parent to children.

A worked split calculation

Say a node has 10 samples: 6 class A and 4 class B. Its Gini impurity is:

Gini = 1 - (0.6² + 0.4²) = 1 - (0.36 + 0.16) = 0.48

Now suppose a split sends {5 A, 0 B} left and {1 A, 4 B} right. The children's impurities are 0 (pure) and 1 - (0.2² + 0.8²) = 0.32. The weighted child impurity is (5/10)*0 + (5/10)*0.32 = 0.16. Information gain = 0.48 − 0.16 = 0.32. The tree keeps the split with the highest such gain.

Implementing it in Python

scikit-learn even prints the tree as text so you can read the rules:

from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42)

# max_depth is the main knob against overfitting.
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(Xtr, ytr)

print("accuracy:", round(accuracy_score(yte, tree.predict(Xte)), 4))
print(export_text(tree, feature_names=load_iris().feature_names))

Output:

accuracy: 1.0
|--- petal length (cm) <= 2.45
|   |--- class: 0
|--- petal length (cm) >  2.45
|   |--- petal length (cm) <= 4.75
|   |   |--- petal width (cm) <= 1.65
|   |   |   |--- class: 1
|   |   |--- petal width (cm) >  1.65
|   |   |   |--- class: 2
|   |--- petal length (cm) >  4.75
|   |   |--- petal width (cm) <= 1.75
|   |   |   |--- class: 1
|   |   |--- petal width (cm) >  1.75
|   |   |   |--- class: 2

Read it top to bottom: if petal length ≤ 2.45 cm it is class 0 (setosa); otherwise further questions on petal length and width separate versicolor (class 1) from virginica (class 2). That is a complete, human-readable model — no black box.

Why trees overfit — and how to stop them

Left unchecked, a tree will keep splitting until every leaf holds a single training example. It then has 100% training accuracy and terrible test accuracy — it has memorised noise. Controls:

• max_depth — cap how many questions deep the tree can go (the single most important knob).
• min_samples_leaf / min_samples_split — refuse to split groups that are already small.
• max_leaf_nodes — cap the total number of leaves.
• Cost-complexity pruning (ccp_alpha) — grow a full tree, then snip back branches that add little.

Preparing your data

Trees are wonderfully low-maintenance:

• No scaling needed. Splits are based on thresholds, so units do not matter.
• Handle categorical features by encoding them (scikit-learn trees need numeric input).
• Missing values should be imputed or handled before training in scikit-learn.
• Watch class imbalance — use class_weight='balanced' so the tree does not ignore the rare class.

Strengths, weaknesses, and what comes next

Strengths: interpretable, fast, handle mixed data types, capture non-linear relationships and interactions automatically.

Weakness: a single tree is unstable — a small change in the data can produce a very different tree, and it easily overfits. The fix is to combine many trees, which leads directly to random forests and gradient boosting.

Key takeaways

• A decision tree is a learned flowchart of yes/no questions.
• Splits are chosen to maximise information gain (drop in Gini or entropy).
• Depth limits and pruning are essential to prevent overfitting.
• Single trees are the building block of the far stronger random forest and boosting ensembles.

Further reading: scikit-learn trees and the visual guide.