Choosing an Optimizer

What Does Choosing an Optimizer Mean?

During training, the neural network calculates gradients using backpropagation.

Input
  ↓
Forward Propagation
  ↓
Prediction
  ↓
Loss
  ↓
Backpropagation
  ↓
Gradients
  ↓
Optimizer
  ↓
Update Weights

The optimizer decides how those gradients should be used to change the weights.

Different optimizers use different strategies for making those updates.

Common Optimizers

Optimizer
│
├── SGD
│
├── Momentum
│
└── Adam

For this course, the important optimizers are: SGD, Momentum, and Adam.

You do not need to memorize every optimizer in existence. First understand how these three behave.

1. SGD

SGD stands for Stochastic Gradient Descent. It updates weights using the gradient and learning rate.

weight =
    weight - learning_rate × gradient

Example:

weight = 0.50
gradient = 0.20
learning_rate = 0.10

new_weight =
    0.50 - (0.10 × 0.20)

new_weight =
    0.48

SGD is simple and gives you direct control over the optimization process.

Its weakness is that training can sometimes be slower or more sensitive to the learning-rate choice.

2. Momentum

Momentum keeps information from previous gradients.

Current Gradient
       +
Previous Gradient Information
       ↓
Momentum
       ↓
Weight Update

Think about pushing a heavy ball downhill.

Without Momentum:

gradient → small movement
gradient → small movement
gradient → small movement


With Momentum:

gradient → movement
             ↓
          momentum
             ↓
          movement
             ↓
          momentum
             ↓
       faster progress

Momentum can help reduce inefficient back-and-forth movement during optimization.

3. Adam

Adam combines momentum-like gradient history with adaptive updates based on gradient magnitude.

Gradient
   ↓
 ┌───────────────┐
 ↓               ↓
First Moment     Second Moment
 ↓               ↓
Direction        Magnitude
 └───────┬───────┘
         ↓
Adaptive Update
         ↓
Weight Update

This often makes Adam a convenient starting point for training neural networks.

A common starting configuration is:

optimizer = Adam(
    learning_rate=0.001
)

But "start with Adam" does not mean "always use Adam."

Simple Comparison

SGD

Gradient
   ↓
Learning Rate
   ↓
Update


Momentum

Gradient
   ↓
Gradient History
   ↓
Momentum
   ↓
Update


Adam

Gradient
   ↓
First Moment + Second Moment
   ↓
Adaptive Update
   ↓
Update

The main difference is how much information each optimizer uses when deciding the next update.

Which Optimizer Should You Start With?

If you are learning deep learning or starting a new neural network, Adam is often a practical first choice.

New Neural Network
        ↓
      Adam
        ↓
Train Model
        ↓
Check Validation Performance

Why?

Adam
│
├── Keeps gradient history
├── Adapts updates
└── Usually requires less manual
    optimizer tuning than basic SGD

This makes it convenient when you are more interested in getting a model training correctly first rather than manually tuning optimization behavior from scratch.

Example 1 — Image Classification

Suppose you are building a neural network that classifies images as:

Cat
Dog
Car
Bird

You have never trained this particular model before.

A reasonable starting point is:

optimizer = Adam(
    learning_rate=0.001
)

Then monitor:

Training Loss
Validation Loss
Training Accuracy
Validation Accuracy

If the model trains well and validation performance is good, there may be no reason to change the optimizer.

Example 2 — Comparing Adam and SGD

Suppose you train the same model twice.

Model A
Optimizer = Adam

Model B
Optimizer = SGD

After training, imagine you get:

Adam

Training Accuracy    = 98%
Validation Accuracy  = 91%


SGD

Training Accuracy    = 96%
Validation Accuracy  = 93%

You should not automatically choose Adam just because its training accuracy is higher.

SGD produced better validation accuracy in this example. That can mean it generalizes better to unseen data.

This is why optimizer selection should be based on validation performance, not just training loss.

Learning Rate Is Often More Important

Choosing Adam instead of SGD does not solve everything. The learning rate is extremely important.

Learning Rate Too Large

Loss
 ↓
 ↓
 ↑
 ↓
 ↑
 ↓
Training becomes unstable

Learning Rate Too Small

Loss
 ↓
 ↓
 ↓
 ↓
 ↓

Training becomes very slow

A reasonable optimizer with a bad learning rate can perform worse than a different optimizer with a good learning rate.

A Simple Decision Process

Start
  ↓
Choose Adam
  ↓
Choose reasonable learning rate
  ↓
Train
  ↓
Check Training Loss
  ↓
Check Validation Loss
  ↓
Is validation performance good?
  │
  ├── Yes
  │     ↓
  │   Keep it
  │
  └── No
        ↓
    Investigate
        ↓
    Tune learning rate
        ↓
    Try another optimizer
        ↓
    Compare results

This is much better than randomly switching between optimizers whenever training looks bad.

When Should You Consider SGD?

SGD is worth trying when you want more direct control over the optimization process or when experiments show that it gives better validation or generalization performance.

Adam
→ Fast/convenient starting point


SGD
→ More manual control
→ Can sometimes generalize differently

Do not assume that a faster decrease in training loss means a better final model.

When Should You Consider Momentum?

Momentum is useful when you want the benefits of accumulated gradient direction while keeping a relatively simple optimization method.

SGD
 ↓
Add Momentum
 ↓
Better use of previous gradient direction

In practice, many modern workflows start with Adam or one of its variants rather than manually choosing plain SGD with Momentum, but Momentum remains important for understanding optimization.

Do Not Choose Only by Training Loss

This is one of the most important points in this lesson.

Model A

Training Loss     = 0.05
Validation Loss   = 0.60


Model B

Training Loss     = 0.10
Validation Loss   = 0.30

Model A has lower training loss, but Model B has much lower validation loss.

If your goal is good performance on unseen data, Model B may be the better model.

Optimizer selection therefore needs to consider validation behavior.

Practical Rule

For a new model:

1. Start with Adam
2. Choose a reasonable learning rate
3. Train the model
4. Monitor training loss
5. Monitor validation loss
6. Check validation performance
7. Tune learning rate if necessary
8. Compare another optimizer if needed

This gives you a controlled experiment instead of guessing.

Choosing an Optimizer With Python

Here is a simple example using TensorFlow/Keras.

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(1, activation="sigmoid")
])

optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)

model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

Here we chose Adam as the optimizer.

optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)

We could change it to SGD:

optimizer = tf.keras.optimizers.SGD(
    learning_rate=0.01
)

Or SGD with Momentum:

optimizer = tf.keras.optimizers.SGD(
    learning_rate=0.01,
    momentum=0.9
)

The rest of the model can remain the same. This makes it possible to compare optimizers experimentally.

Complete Comparison Example

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(
        16,
        activation="relu",
        input_shape=(4,)
    ),

    tf.keras.layers.Dense(
        1,
        activation="sigmoid"
    )
])

# Try Adam
optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)

model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

history = model.fit(
    X_train,
    y_train,
    validation_data=(X_val, y_val),
    epochs=20
)

After training, inspect the validation results.

validation_loss
validation_accuracy

Then you can repeat the experiment with SGD and compare the results.

Important: Do Not Chase the "Best" Optimizer

There is no optimizer that wins on every dataset and every neural network.

Dataset A
→ Adam may work better


Dataset B
→ SGD may work better


Dataset C
→ Another optimizer may work better

The correct question is not:

"Which optimizer is the best?"

The better question is:

"Which optimizer gives the best
validation performance for my model and data?"

Remember This

SGD
→ Simple
→ Direct gradient updates


Momentum
→ Uses previous gradient direction
→ Helps smooth optimization


Adam
→ Uses first moment
→ Uses second moment
→ Adaptive updates
→ Good practical starting point

A simple strategy is:

New Model
   ↓
Start with Adam
   ↓
Monitor validation performance
   ↓
Tune learning rate
   ↓
Compare SGD / Momentum if useful
   ↓
Keep the optimizer that works best
for your actual problem