What Is an Optimizer?

What Is an Optimizer?

A neural network learns by changing its weights. But the important question is: how should those weights be changed?

An optimizer answers that question.

Neural Network
       ↓
Prediction
       ↓
Loss
       ↓
Backpropagation
       ↓
Gradients
       ↓
Optimizer
       ↓
Updated Weights
       ↓
Better Prediction

So an optimizer is an algorithm that determines how the model's parameters should be updated during training.

Why Do We Need an Optimizer?

Suppose a neural network has a weight:

weight = 0.50

The model makes a prediction, calculates the loss, and backpropagation gives us a gradient:

gradient = 0.20

We now know that the weight needs to change, but we still need a rule for deciding how much to change it.

That is where the optimizer comes in.

Gradient
   ↓
Optimizer
   ↓
How should the weight change?
   ↓
Updated Weight

Simple Example

Suppose we have:

Old Weight    = 0.50
Gradient      = 0.20
Learning Rate = 0.10

A basic gradient descent optimizer uses:

new_weight =
    old_weight - learning_rate × gradient

Substitute the values:

new_weight
=
0.50 - (0.10 × 0.20)

=
0.50 - 0.02

=
0.48

The optimizer changes the weight from 0.50 to 0.48.

Old Weight
   0.50
     ↓
  Optimizer
     ↓
Updated Weight
   0.48

Gradient vs Optimizer

These two concepts are related, but they are not the same thing.

Gradient
→ Tells us how the loss changes
  with respect to a weight.


Optimizer
→ Uses that gradient to update
  the weight.

Think of the gradient as information and the optimizer as the rule that acts on that information.

Gradient
"What direction should I move?"

        ↓

Optimizer
"How should I use that information
to update the weight?"

Easy Mountain Example

Imagine you are standing on a mountain and want to reach the lowest point.

          Mountain
             /\
            /  \
           /    \
          /  ●   \
         /        \
        /__________\
             ↓
          Lowest Point

The goal of training is similar: we want to move toward a point where the loss is lower.

High Loss
    ↓
Move toward lower loss
    ↓
Lower Loss
    ↓
Repeat
    ↓
Very Low Loss

The gradient gives information about the slope, while the optimizer determines how to use that information to move.

Optimizer and Learning Rate

A basic optimizer often works together with a learning rate.

learning_rate = 0.1

The learning rate controls the size of the update.

Small Learning Rate
        ↓
Small Updates


Large Learning Rate
        ↓
Large Updates

For example:

Gradient = 0.5


Learning Rate = 0.01

Update = 0.01 × 0.5
       = 0.005


Learning Rate = 0.1

Update = 0.1 × 0.5
       = 0.05

The learning rate is therefore an important setting that affects how aggressively the optimizer changes the weights.

What If the Learning Rate Is Too Small?

The model may learn very slowly.

Weight
  ↓
Tiny Update
  ↓
Tiny Update
  ↓
Tiny Update
  ↓
Very Slow Learning

For example:

Learning Rate = 0.00001

Weight:
0.500000
0.499995
0.499990
0.499985
...

The updates may be so small that training takes a very long time.

What If the Learning Rate Is Too Large?

The optimizer may make updates that are too large.

Loss
 ↓
Big Update
 ↓
Overshoot
 ↓
Big Update
 ↓
Overshoot Again
 ↓
Unstable Training

Instead of moving smoothly toward a low-loss region, the model can jump around or even diverge.

So the goal is not simply "use the biggest learning rate." The update size needs to be appropriate for the problem.

A Simple Optimizer in Python

We can implement the basic gradient descent update ourselves using Python.

weight = 0.50
gradient = 0.20
learning_rate = 0.10

weight = weight - learning_rate * gradient

print(weight)

Output:

0.48

This is the basic idea behind gradient descent.

Creating a Simple Optimizer Function

We can make the update logic into a function:

def update_weight(weight, gradient, learning_rate):

    new_weight = weight - learning_rate * gradient

    return new_weight


weight = 0.50
gradient = 0.20
learning_rate = 0.10

weight = update_weight(
    weight,
    gradient,
    learning_rate
)

print(weight)

Output:

0.48

The function receives the current weight, gradient, and learning rate, then returns the updated weight.

Optimizer Repeats the Process

Neural networks do not update their weights only once. The process happens repeatedly.

weight = 0.50
learning_rate = 0.10

gradient = 0.20
weight = weight - learning_rate * gradient

print(weight)

gradient = 0.10
weight = weight - learning_rate * gradient

print(weight)

gradient = 0.05
weight = weight - learning_rate * gradient

print(weight)

The simplified output is:

0.48
0.47
0.465

The weight changes again and again as the model learns.

Are All Optimizers the Same?

No.

Different optimizers use different strategies for updating the model's parameters.

Gradient Descent
→ Basic approach


SGD
→ Updates using individual training examples


Mini-Batch Gradient Descent
→ Updates using small batches


Momentum
→ Uses previous update information


Adam
→ Adapts updates using gradient statistics

We will study each of these in the following topics.

Where the Optimizer Fits

Training Data
      ↓
Neural Network
      ↓
Prediction
      ↓
Loss Function
      ↓
Loss
      ↓
Backpropagation
      ↓
Gradients
      ↓
Optimizer
      ↓
Updated Weights
      ↓
Next Training Step

This is the connection between the concepts from the previous lessons and the optimizer.

Forward Propagation
        ↓
Calculate Loss
        ↓
Backpropagation
        ↓
Calculate Gradients
        ↓
Optimizer
        ↓
Update Weights
        ↓
Repeat

Simple Real-World Example

Imagine you are adjusting the temperature of a shower.

Water too cold
     ↓
Increase temperature

Water still cold
     ↓
Increase again

Water too hot
     ↓
Decrease temperature

The goal is to find a comfortable temperature.

Similarly, a neural network changes its weights repeatedly to find parameter values that produce lower loss.

High Loss
   ↓
Adjust Weights
   ↓
Calculate Loss Again
   ↓
Adjust Weights Again
   ↓
Lower Loss

The analogy is not mathematically exact, but it is useful for understanding the basic idea of iterative optimization.

Do Not Confuse These Concepts

Loss Function
→ Measures how wrong the prediction is.


Backpropagation
→ Calculates gradients of the loss.


Gradient
→ Describes how the loss changes
  with respect to a parameter.


Optimizer
→ Uses gradients to update parameters.


Learning Rate
→ Controls the size of those updates.

These concepts work together, but they have different jobs.

Remember This

Optimizer
=
Algorithm that updates
neural network parameters.


Its basic job:

Gradient
   ↓
Optimizer
   ↓
Updated Weight
   ↓
Lower Loss
   ↓
Repeat

The most important sentence to remember is: an optimizer uses the gradients produced by backpropagation to decide how the neural network's weights should be updated.