Why Do We Split the Data?
A Machine Learning model should not simply memorize the examples it has already seen. We need to know whether it can work well on new examples it has never seen before.
We split data so we can train the model and honestly check how well it generalizes.
The basic idea is simple: use some examples to learn, and keep other examples separate so we can evaluate what the model learned.
Imagine Studying for an Exam
Suppose you have an exam tomorrow.
You practice 100 questions and memorize all the answers.
Then your teacher gives you exactly those same 100 questions in the exam.
Not necessarily. You may simply remember the answers to the questions you already practiced.
A better test would contain questions you have not seen before.
Machine Learning Has the Same Problem
A Machine Learning model can also perform extremely well on examples it has already seen.
But that does not automatically mean it will perform well on new data.
Examples used during learning.
Learns patterns from the examples.
Examples it has never seen.
We care about the model's performance on new data, not just the data it memorized during training.
What Does "Generalize" Mean?
Generalization means that a model can use what it learned from its training examples to make useful predictions on new examples.
We do not want a model that only remembers the training data. We want a model that learns useful patterns that continue to work on unseen data.
A House Price Example
Suppose we have 10,000 historical house records.
Each sample contains:
- Size.
- Bedrooms.
- Bathrooms.
- Age.
- Actual selling price.
We could give all 10,000 examples to the model and then measure its performance on those exact same examples.
That would be a bad evaluation strategy.
Why Testing on Training Data Is Misleading
Imagine the model predicts the price of every house in the training dataset very accurately.
The model has already seen these examples during training.
The model struggles when it encounters examples it has never seen.
If we only looked at training performance, we might incorrectly believe the model was excellent.
So We Keep Some Data Aside
Instead of using every sample for training, we divide the dataset into separate parts.
Used to train the model.
Used while developing the model.
Used for final evaluation.
The Three Parts Have Different Jobs
The model learns patterns from these examples.
Helps us compare choices while developing the model.
Provides a final check using unseen examples.
Why Not Just Use Two Parts?
You can sometimes use a simple training/test split, especially for smaller or straightforward projects.
But during model development, we often make decisions about the model based on evaluation results.
If we repeatedly use the final test data to make those decisions, the test set is no longer a truly independent final check.
If you keep changing your model based on test results, you gradually start optimizing for the test set itself.
A Simple Mental Model
Learn from examples.
Make development decisions.
Perform the final evaluation.
The Important Question
Whenever you build a Machine Learning model, don't only ask:
Ask the more important question:
Splitting the data gives us a practical way to answer that question.
Don't Evaluate a Model Only on What It Already Saw
Separate parts of the dataset allow us to distinguish learning from evaluation and give us a better idea of how the model may perform on unseen examples.
Why Split the Data?
Suppose you have 10,000 house samples and your model predicts all 10,000 training prices with extremely low error.
Does that alone prove that the model will perform well on houses it has never seen?
The model may have memorized patterns specific to the training examples. We need separate unseen data to evaluate whether it generalizes.