Overfitting in Decision Trees
A Decision Tree can become too complicated and learn the training data too closely. This is called overfitting.
A tree can learn the training data too well.
When a Decision Tree keeps making more and more splits, it can start learning small details and noise that do not represent useful patterns. The result may be excellent on training data but poor on new data.
What Is Overfitting?
Overfitting happens when a Machine Learning model learns the training examples too closely instead of learning general patterns.
With a Decision Tree, this usually happens when the tree becomes very deep and creates many small branches.
Makes reasonable decisions that can work on new data.
Matches training examples extremely closely but may fail on new data.
A Simple Student Example
Suppose we want to predict whether a student will Pass or Fail.
We have training data containing information such as:
A simple tree might learn a useful pattern such as:
What Happens When the Tree Gets Too Deep?
Instead of stopping after learning useful patterns, the tree may continue splitting the data again and again.
It may create rules that are extremely specific to individual training examples.
If the tree keeps splitting until it almost perfectly separates every training example, it can become too complicated.
Memorizing Instead of Learning
This is the easiest way to understand overfitting.
Imagine a student memorizes the exact answers from a practice exam instead of learning the underlying topic.
Can solve new questions that use the same concept.
Performs well on familiar questions but struggles with new ones.
An overfitted Decision Tree behaves similarly. It remembers the training examples too closely instead of learning patterns that generalize.
Training Performance vs New Data
One of the clearest signs of overfitting is the difference between performance on training data and unseen data.
The tree has learned the training examples extremely closely.
The tree struggles when it sees examples it did not memorize.
Why Does a Deep Tree Overfit?
A Decision Tree can keep creating more specific rules as it grows deeper.
Learns broad patterns.
Learns more specific patterns.
Can memorize training data.
More complexity is not automatically better.
A Simple Example of Overfitting
Imagine the training data contains a student who studied 5 hours and passed.
An overly complex tree might eventually create a rule that is effectively specific to that training example.
→ Pass
This rule may perfectly describe that training example, but Student ID = 1042 does not tell us anything useful about whether another student will pass.
How Do We Recognize Overfitting?
A common warning sign is a large gap between training performance and performance on unseen data.
New data: 89%
New data: 65%
The exact numbers are only an example. The important idea is the gap between training performance and performance on unseen data.
Overfitting vs Underfitting
These are opposite problems.
The model has not learned enough from the data.
The model performs reasonably well on both training and new data.
The model learns the training data too closely.
How Can We Reduce Overfitting?
The basic idea is to prevent the tree from becoming unnecessarily complex.
In Python, Decision Tree libraries provide parameters that can control the complexity of the tree.
We will use those parameters when we build a Decision Tree with Python later in this lesson.
The Real Goal
The goal is not to create the tree with the highest possible training accuracy.
The goal is to create a model that can make good predictions on data it has never seen before.
A Decision Tree should learn patterns, not memorize examples.
If the tree becomes too deep and specific, it may fit the training data extremely well while performing poorly on new data. That problem is called overfitting.