Training vs Validation Loss

What Is Training Loss?

During training, the neural network makes predictions on training data and calculates the loss.

Training Data
      ↓
Neural Network
      ↓
Prediction
      ↓
Training Loss
      ↓
Backpropagation
      ↓
Update Weights

The training loss tells us how wrong the model is on the data it is using to learn.

As training progresses, we generally want the training loss to decrease.

Epoch 1 → Loss = 1.20
Epoch 2 → Loss = 0.80
Epoch 3 → Loss = 0.55
Epoch 4 → Loss = 0.35
Epoch 5 → Loss = 0.22

The decreasing values indicate that the model is becoming better at fitting the training data.

What Is Validation Loss?

Validation loss is calculated using a separate validation dataset.

This data is not used to directly update the model's weights during the training step.

Validation Data
      ↓
Neural Network
      ↓
Prediction
      ↓
Validation Loss

The purpose is to check whether the model is performing well on data it did not use to learn its weights.

This gives us a better idea of how well the model may generalize to unseen data.

Simple Example

Suppose we have 1,000 images of cats and dogs.

Total Images = 1,000

Training Data
→ 800 images

Validation Data
→ 200 images

The model learns its weights using the training data.

800 Training Images
        ↓
    Train Model
        ↓
   Update Weights

The validation images are used separately to check how well the current model performs.

200 Validation Images
        ↓
   Test Current Model
        ↓
 Calculate Validation Loss

The validation data helps us detect whether the model is simply memorizing the training examples instead of learning patterns that generalize.

Why Do We Need Both?

Looking only at training loss is not enough.

A model can become very good at the training data while becoming worse at handling new data.

Training Loss
→ How well am I learning the training data?

Validation Loss
→ How well am I performing on separate data?

Looking at both values gives us a much better picture of what is happening during training.

When Training and Validation Loss Both Decrease

This is generally a good sign.

Epoch     Training Loss     Validation Loss

1             1.00                1.10
2             0.70                0.82
3             0.50                0.61
4             0.35                0.45
5             0.25                0.34

Both losses are going down.

Training Loss
1.00
 ↓
0.70
 ↓
0.50
 ↓
0.35
 ↓
0.25


Validation Loss
1.10
 ↓
0.82
 ↓
0.61
 ↓
0.45
 ↓
0.34

This suggests that the model is improving on both its training data and its separate validation data.

When Training Loss Decreases but Validation Loss Increases

This is one of the most important patterns to recognize.

Epoch     Training Loss     Validation Loss

1             1.00                1.10
2             0.70                0.80
3             0.45                0.60
4             0.25                0.72
5             0.12                0.95

Notice what happens after epoch 3.

Training Loss
0.45
 ↓
0.25
 ↓
0.12

Getting lower


Validation Loss
0.60
 ↓
0.72
 ↓
0.95

Getting higher

This is a classic sign of overfitting.

The model is continuing to fit the training data, but its performance on validation data is getting worse.

Easy Example of Overfitting

Imagine a student preparing for an exam by memorizing the exact answers from practice questions.

Practice Questions
        ↓
Memorize Answers
        ↓
Very Good Practice Score

But when the teacher gives different questions, the student performs poorly.

New Questions
      ↓
Poor Performance

That is similar to what can happen with an overfitted neural network.

Training Data
→ Model performs very well

New / Validation Data
→ Model performs worse

The model learned the training data too specifically instead of learning patterns that generalize well.

What About Underfitting?

The opposite problem is called underfitting.

The model has not learned enough useful patterns from the training data.

Training Loss
→ High

Validation Loss
→ High

For example:

Epoch     Training Loss     Validation Loss

1             1.50                1.60
2             1.30                1.45
3             1.20                1.35
4             1.15                1.30
5             1.10                1.25

Both losses remain relatively high. This can indicate that the model is not learning enough from the data, although the exact cause needs further investigation.

Three Important Patterns

Pattern 1

Training Loss     ↓
Validation Loss   ↓

→ Generally good learning


Pattern 2

Training Loss     ↓
Validation Loss   ↑

→ Possible overfitting


Pattern 3

Training Loss     High
Validation Loss   High

→ Possible underfitting

These patterns are useful signals, but they should not be treated as automatic diagnoses. Other issues such as data quality, distribution differences, optimization settings, or an unsuitable model can also affect the losses.

Training and Validation Loss With Python

A simplified training process can look like this:

for epoch in range(5):

    # Train
    prediction = model(train_data)

    training_loss = calculate_loss(
        prediction,
        train_target
    )

    gradients = calculate_gradients(
        training_loss
    )

    update_weights(gradients)

    # Validate
    validation_prediction = model(
        validation_data
    )

    validation_loss = calculate_loss(
        validation_prediction,
        validation_target
    )

    print("Epoch:", epoch + 1)
    print("Training Loss:", training_loss)
    print("Validation Loss:", validation_loss)

Notice that the model updates its weights using the training loss.

Training Loss
     ↓
Gradients
     ↓
Update Weights

Validation loss is calculated to evaluate the current model; it is not used as the ordinary training update signal.

Validation Data
      ↓
Prediction
      ↓
Validation Loss
      ↓
Evaluate Model

Tracking Loss During Training

We normally record training and validation loss after each epoch.

Epoch 1
Training Loss   = 1.20
Validation Loss = 1.15

Epoch 2
Training Loss   = 0.80
Validation Loss = 0.82

Epoch 3
Training Loss   = 0.55
Validation Loss = 0.60

Epoch 4
Training Loss   = 0.40
Validation Loss = 0.48

Epoch 5
Training Loss   = 0.30
Validation Loss = 0.42

Keeping these values allows us to observe whether the model is improving or whether the training behavior is becoming problematic.

Why Validation Loss Is Important

Suppose validation loss reaches its lowest value at epoch 4, but then starts increasing.

Epoch     Validation Loss

1             1.10
2             0.80
3             0.60
4             0.45   ← Lowest
5             0.52
6             0.70
7             0.90

Epoch 4 may represent a better-performing point for generalization than later epochs.

In practice, techniques such as early stopping can monitor validation performance and stop training when continued training is no longer improving the chosen validation metric.

Training Data vs Validation Data

Dataset
   │
   ├── Training Data
   │       ↓
   │    Learn Weights
   │
   └── Validation Data
           ↓
       Evaluate Model

The important point is that the validation data should remain separate from the data used to update the model's weights during training.

Otherwise, the validation measurement can become less useful as an independent check of generalization.

Complete Example

Imagine we train a neural network for 6 epochs.

Epoch     Training Loss     Validation Loss

1             1.20                1.10
2             0.85                0.78
3             0.60                0.55
4             0.42                0.40
5             0.28                0.47
6             0.18                0.65

What can we observe?

Epoch 1 → Both losses decrease
Epoch 2 → Both losses decrease
Epoch 3 → Both losses decrease
Epoch 4 → Validation loss reaches a low point
Epoch 5 → Training improves, validation worsens
Epoch 6 → Training improves, validation worsens more

The model continues improving on the training data, but its validation performance has started getting worse.

This is a strong signal that the model may be starting to overfit.

Easy Way to Remember

Training Loss
=
"How well am I doing on the data I am learning from?"


Validation Loss
=
"How well am I doing on separate data?"

Think of it like studying for an exam:

Training Data
→ Practice Questions

Validation Data
→ Different Questions

Training Loss
→ Practice Performance

Validation Loss
→ Performance on Different Questions

If you only get better at the practice questions but worse at different questions, you may be memorizing instead of learning general patterns.

Remember This

Training Loss
→ Measures loss on training data
→ Used during the training process


Validation Loss
→ Measures loss on separate validation data
→ Used to evaluate generalization


Both Decrease
→ Usually a healthy sign


Training ↓
Validation ↑
→ Possible overfitting


Both Remain High
→ Possible underfitting

The most important idea is: training loss tells you how well the model is learning the training data, while validation loss helps you determine whether that learning generalizes to separate data.