Improve the Model

Why Do We Improve a Model?

Our first model may not perform as well as we want.

First Model

Training Accuracy = 90%
Test Accuracy     = 87%

        ↓

Need improvement

We can try different approaches to improve its ability to learn useful patterns and generalize to unseen data.

First Find the Problem

Before changing the model, compare the training and validation or test performance.

Training Accuracy = 99%
Test Accuracy     = 85%

        ↓

Possible Overfitting

Another situation:

Training Accuracy = 70%
Test Accuracy     = 68%

        ↓

Possible Underfitting

The solution depends on the problem.

Improve the Data

Better data can often improve a model more than simply making the neural network larger.

For image problems, one approach is data augmentation.

Original Image
      ↓
Rotate slightly
      ↓
Shift
      ↓
Zoom
      ↓
Flip
      ↓
New training examples

The model gets more variation in the training data and can become less dependent on the exact appearance of the original images.

Data Augmentation With Python

data_augmentation = tf.keras.Sequential([
    tf.keras.layers.RandomRotation(0.1),
    tf.keras.layers.RandomZoom(0.1),
])

These layers randomly transform training images.

Original
   ↓
Random Rotation
   ↓
Random Zoom
   ↓
Modified Training Image

This can help the model learn patterns that are less sensitive to small changes in the input.

Add More Neurons

A model that is too small may not have enough capacity to learn the patterns in the data.

For example, we could change:

Dense(64, activation="relu")

to:

Dense(128, activation="relu")

This gives that layer more neurons and therefore more parameters to learn from the data.

Add Another Layer

We can also increase the depth of the network.

model = tf.keras.Sequential([

    tf.keras.layers.Flatten(
        input_shape=(28, 28)
    ),

    tf.keras.layers.Dense(
        128,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        64,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        32,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        10,
        activation="softmax"
    )
])

The extra hidden layer gives the network another level at which it can transform the learned representation.

Be Careful With Making the Model Bigger

Bigger does not automatically mean better.

Small Model
     ↓
May underfit

Very Large Model
     ↓
May overfit

Right-sized Model
     ↓
Better generalization

A larger network also means more parameters, more computation, and potentially a greater risk of overfitting.

Increase the Number of Epochs

If the model has not trained enough, we can increase the number of epochs.

model.fit(
    x_train,
    y_train,
    epochs=10
)

Compared with:

epochs=5

The model now gets more opportunities to update its weights.

But this is not a free improvement. If the model starts overfitting, more epochs can make the test or validation performance worse.

Change the Batch Size

The batch size controls how many training examples are processed before an update.

batch_size=32

means the model processes 32 examples at a time.

We can experiment with another value:

batch_size=64

Batch size can affect training speed, memory usage, and optimization behavior. It is a hyperparameter that can be tuned rather than a guaranteed accuracy improvement.

Change the Learning Rate

The learning rate controls how large the weight updates are during optimization.

Learning Rate

Too Large
    ↓
Training may become unstable

Too Small
    ↓
Training may become very slow

Good Value
    ↓
Steady learning

For example, with Adam:

optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)

We can experiment with a different learning rate if the training behavior suggests it is appropriate.

Use Dropout to Reduce Overfitting

Dropout randomly disables some neurons during training.

model = tf.keras.Sequential([

    tf.keras.layers.Flatten(
        input_shape=(28, 28)
    ),

    tf.keras.layers.Dense(
        128,
        activation="relu"
    ),

    tf.keras.layers.Dropout(0.2),

    tf.keras.layers.Dense(
        64,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        10,
        activation="softmax"
    )
])

With Dropout(0.2), approximately 20% of the layer's outputs are randomly dropped during training.

This can make the network less dependent on particular neurons and can help reduce overfitting.

Use Early Stopping

Sometimes the model improves for a while and then starts getting worse on validation data.

Epoch 1 → Validation accuracy = 91%
Epoch 2 → Validation accuracy = 94%
Epoch 3 → Validation accuracy = 96%
Epoch 4 → Validation accuracy = 97%
Epoch 5 → Validation accuracy = 96%
Epoch 6 → Validation accuracy = 95%

We could stop training when validation performance stops improving.

early_stopping = tf.keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=2,
    restore_best_weights=True
)

model.fit(
    x_train,
    y_train,
    epochs=20,
    validation_split=0.1,
    callbacks=[early_stopping]
)

Here, training can stop when validation loss fails to improve for the specified number of epochs.

Compare Training and Validation Results

The training history helps us understand whether the model is improving or overfitting.

history = model.fit(
    x_train,
    y_train,
    epochs=10,
    validation_split=0.1
)

We can inspect the recorded accuracy:

history.history["accuracy"]

history.history["val_accuracy"]

For example:

Training:
95% → 97% → 99%

Validation:
94% → 96% → 95%

The gap between training and validation performance is a useful warning sign.

Example: Fixing Overfitting

Suppose our original model gives:

Training Accuracy = 99%
Validation Accuracy = 86%

Instead of simply adding more layers, we could try regularization such as dropout and data augmentation.

Data Augmentation
        +
Dropout
        +
Early Stopping
        ↓
Retrain
        ↓
Evaluate Again

The goal is not to make training accuracy as high as possible. The goal is to improve performance on unseen data.

Example: Fixing Underfitting

Suppose:

Training Accuracy = 70%
Validation Accuracy = 68%

The model is struggling even with its training data.

Possible approaches include:

More training epochs
        ↓
or
        ↓
More neurons
        ↓
or
        ↓
Additional layers
        ↓
or
        ↓
Better input/data preparation

The correct choice depends on why the model is underfitting.

A Better Model Example

import tensorflow as tf

model = tf.keras.Sequential([

    tf.keras.layers.Input(
        shape=(28, 28)
    ),

    tf.keras.layers.Flatten(),

    tf.keras.layers.Dense(
        128,
        activation="relu"
    ),

    tf.keras.layers.Dropout(0.2),

    tf.keras.layers.Dense(
        64,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        10,
        activation="softmax"
    )
])

model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

This model introduces dropout as one simple technique for improving generalization.

Train the Improved Model

early_stopping = tf.keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=2,
    restore_best_weights=True
)

history = model.fit(
    x_train,
    y_train,
    epochs=20,
    batch_size=32,
    validation_split=0.1,
    callbacks=[early_stopping]
)

Notice that we allow up to 20 epochs, but early stopping can stop training earlier if validation loss stops improving.

Evaluate the Improved Model

test_loss, test_accuracy = model.evaluate(
    x_test,
    y_test
)

print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

Now compare this result with the original model.

Original Model
Test Accuracy = 94%

Improved Model
Test Accuracy = 96%

If the improvement is consistent and meaningful, the new model may be a better choice.

Improvement Is an Experiment

A common mistake is changing many things at once.

Change:
- layers
- neurons
- batch size
- learning rate
- dropout
- epochs

All at once

        ↓

Hard to know what helped

A better approach is to make controlled changes.

Baseline Model
      ↓
Change ONE important thing
      ↓
Train
      ↓
Evaluate
      ↓
Compare
      ↓
Keep or reject the change

Complete Improvement Workflow

Build Model
     ↓
Train
     ↓
Evaluate
     ↓
Check Results
     ↓
Is there a problem?
     ↓
Yes
     ↓
Identify Problem
     ↓
Choose Improvement
     ↓
Change Model / Data / Training
     ↓
Train Again
     ↓
Evaluate Again
     ↓
Compare Results

This cycle is one of the most important ideas in practical machine learning.

Quick Check