Training vs Validation vs Test
Training, validation, and test data all come from the dataset, but each has a different job in the Machine Learning workflow.
Learn with training data, make development decisions with validation data, and perform the final evaluation with test data.
Keeping these roles separate helps us understand whether a model can generalize beyond the examples used during development.
The Three Datasets at a Glance
Used by the model to learn patterns from examples.
Used during development to compare approaches and make decisions.
Used for the final evaluation of the selected model.
Think of Them as Three Different Jobs
Teach the model using known examples.
Compare and improve the model during development.
Measure the final model on held-out examples.
A Concrete Example
Suppose we have 10,000 labeled house records.
For learning purposes, imagine we split them like this:
The percentages shown here are only an example. Real projects may use different strategies depending on the dataset and problem.
Step 1: Training
The model first receives the training data.
The model learns relationships from the training examples.
For a house-price problem, the model might learn relationships involving size, bedrooms, location, age, and other available features.
Step 2: Validation
After training, we can evaluate the model on the validation data.
Use validation results to decide which model or approach should move forward.
We might discover that one model performs better than another on the validation examples.
Step 3: Final Test
Once we have finished making development decisions, we evaluate the selected model using the test data.
Measure the final model on data kept separate from the development process.
What Happens If We Mix Them Up?
The roles are not interchangeable.
Training on the test data makes the test result less meaningful.
Repeatedly changing the model based on the test result turns the test set into part of development.
The Most Important Difference
A Student Analogy
Think about learning for an exam.
You learn concepts and practice problems.
You check your weaknesses and adjust your preparation.
You are evaluated on questions you did not use for preparation decisions.
What If We Only Have Training and Test Data?
You do not always need a separate validation set.
For simple projects, a training/test split can be perfectly reasonable.
You might train a model on 80% of the data and reserve 20% for final evaluation.
The important principle is not blindly following a percentage. The important principle is keeping the final evaluation independent from model development.
The Complete Workflow
Gather examples relevant to the problem.
Separate training, validation, and test data when appropriate.
Learn patterns from training data.
Compare and improve the model.
Perform the final evaluation.
Quick Comparison
| Dataset | Main Purpose | Used During Training? | Used for Final Evaluation? |
|---|---|---|---|
| Training | Learn patterns | Yes | No |
| Validation | Compare and improve | No | No |
| Test | Final evaluation | No | Yes |
Remember the Three Words
Training = Learn.
Validation = Choose.
Test = Evaluate.
Which Dataset?
You have trained three different models and now want to decide which one should be used.
Which dataset should normally help you make that development decision?
Validation data.
After selecting the final approach, the test data should be used for the final evaluation.