Optimizers With Python

Where Does the Optimizer Fit?

Remember the training process:

Input
  ↓
Forward Propagation
  ↓
Prediction
  ↓
Loss
  ↓
Backpropagation
  ↓
Gradients
  ↓
Optimizer
  ↓
Update Weights
  ↓
Next Training Step

The optimizer is responsible for using the gradients to update the neural network's weights.

In TensorFlow/Keras, we usually specify the optimizer when calling model.compile().

Start With a Simple Neural Network

First, create a simple neural network:

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(
        8,
        activation="relu",
        input_shape=(2,)
    ),

    tf.keras.layers.Dense(
        1,
        activation="sigmoid"
    )
])

This model has:

Input
  ↓
Hidden Layer
8 neurons
ReLU
  ↓
Output Layer
1 neuron
Sigmoid

Now we need to tell the model how its weights should be updated. That is where the optimizer comes in.

Using Adam

Adam is one of the most commonly used optimizers for neural networks.

optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)

Then pass the optimizer to compile():

model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

You can also write it directly:

model.compile(
    optimizer=tf.keras.optimizers.Adam(
        learning_rate=0.001
    ),
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

Both approaches do the same thing.

Training the Model With Adam

Suppose we have binary classification data:

X_train = [
    [0, 0],
    [0, 1],
    [1, 0],
    [1, 1]
]

y_train = [
    0,
    1,
    1,
    1
]

We can train the model:

model.fit(
    X_train,
    y_train,
    epochs=100
)

During each training step, Keras approximately performs:

Prediction
    ↓
Calculate Loss
    ↓
Calculate Gradients
    ↓
Adam uses Gradients
    ↓
Update Weights
    ↓
Repeat

Using SGD

SGD is another optimizer.

optimizer = tf.keras.optimizers.SGD(
    learning_rate=0.01
)

Then:

model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

The important part is:

learning_rate=0.01

The learning rate controls how large the weight updates are.

Small learning rate
→ Small weight updates


Large learning rate
→ Large weight updates

Using SGD With Momentum

SGD can also use momentum.

optimizer = tf.keras.optimizers.SGD(
    learning_rate=0.01,
    momentum=0.9
)

Here:

learning_rate=0.01
→ Controls update size


momentum=0.9
→ Uses information from previous updates

So the optimizer can continue moving in a useful direction instead of treating every gradient completely independently.

Compare the Optimizers

# Adam

optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)


# SGD

optimizer = tf.keras.optimizers.SGD(
    learning_rate=0.01
)


# SGD + Momentum

optimizer = tf.keras.optimizers.SGD(
    learning_rate=0.01,
    momentum=0.9
)

Notice something important: the neural network itself does not have to change.

Same Model
    +
Different Optimizer
    ↓
Different Training Behavior

Complete Example With Adam

import tensorflow as tf

# Training data
X_train = tf.constant([
    [0.0, 0.0],
    [0.0, 1.0],
    [1.0, 0.0],
    [1.0, 1.0]
])

y_train = tf.constant([
    0.0,
    1.0,
    1.0,
    1.0
])


# Create model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(
        8,
        activation="relu",
        input_shape=(2,)
    ),

    tf.keras.layers.Dense(
        1,
        activation="sigmoid"
    )
])


# Create optimizer
optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)


# Compile model
model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy"]
)


# Train model
history = model.fit(
    X_train,
    y_train,
    epochs=100,
    verbose=0
)


# Make prediction
prediction = model.predict(
    [[1.0, 1.0]],
    verbose=0
)

print(prediction)

The important optimizer part is:

optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)

And then:

model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

What Happens During model.fit()?

When you run:

model.fit(
    X_train,
    y_train,
    epochs=100
)

Keras performs the training process repeatedly.

Epoch
  ↓
Take training data
  ↓
Forward Pass
  ↓
Prediction
  ↓
Calculate Loss
  ↓
Backpropagation
  ↓
Calculate Gradients
  ↓
Optimizer
  ↓
Update Weights
  ↓
Next Batch
  ↓
Repeat

The optimizer is therefore not responsible for the entire training process. It specifically handles the weight update step.

Understanding Learning Rate in Python

Consider two Adam optimizers:

optimizer1 = tf.keras.optimizers.Adam(
    learning_rate=0.001
)

optimizer2 = tf.keras.optimizers.Adam(
    learning_rate=0.01
)

The second optimizer has a learning rate ten times larger.

0.01 ÷ 0.001 = 10

That does not mean the model will necessarily train ten times faster. It means the optimizer is configured to take substantially larger parameter-update steps, all else equal.

If the learning rate is too large, training can become unstable.

If it is too small, training can become unnecessarily slow.

Example: Change Only the Optimizer

This is a useful experiment because everything else stays the same.

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(
        8,
        activation="relu",
        input_shape=(2,)
    ),

    tf.keras.layers.Dense(
        1,
        activation="sigmoid"
    )
])


# Try Adam
model.compile(
    optimizer=tf.keras.optimizers.Adam(
        learning_rate=0.001
    ),
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

history_adam = model.fit(
    X_train,
    y_train,
    epochs=100,
    verbose=0
)

Now you could create another identical model and use SGD:

model_sgd = tf.keras.Sequential([
    tf.keras.layers.Dense(
        8,
        activation="relu",
        input_shape=(2,)
    ),

    tf.keras.layers.Dense(
        1,
        activation="sigmoid"
    )
])


model_sgd.compile(
    optimizer=tf.keras.optimizers.SGD(
        learning_rate=0.01
    ),
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

history_sgd = model_sgd.fit(
    X_train,
    y_train,
    epochs=100,
    verbose=0
)

Now you can compare their training histories.

Checking the Training Loss

Keras stores the training history in the object returned by model.fit().

history = model.fit(
    X_train,
    y_train,
    epochs=100
)

print(history.history["loss"])

This gives you the loss recorded during each epoch.

Epoch 1   → loss = 0.69
Epoch 2   → loss = 0.63
Epoch 3   → loss = 0.57
...
Epoch 100 → loss = 0.08

A generally decreasing loss indicates that the model is learning from the training data, although validation performance is also important.

Using an Optimizer Manually

Normally Keras hides the gradient calculation and weight update inside model.fit().

But you can also use an optimizer directly.

import tensorflow as tf

x = tf.Variable(5.0)

optimizer = tf.keras.optimizers.SGD(
    learning_rate=0.1
)

with tf.GradientTape() as tape:

    loss = (x - 2) ** 2

gradient = tape.gradient(loss, x)

optimizer.apply_gradients([
    (gradient, x)
])

print(x.numpy())

Here, the goal is to make:

x → 2

The loss is:

loss = (x - 2)²

Since:

x = 5

loss = (5 - 2)²
     = 9

The gradient tells us which direction to move x. The optimizer then applies that gradient to update x.

Understanding apply_gradients()

This line is very important:

optimizer.apply_gradients([
    (gradient, x)
])

It means:

gradient
    ↓
optimizer
    ↓
update x

In a real neural network, there are many weights:

weight_1
weight_2
weight_3
weight_4
...
weight_n

The optimizer applies the appropriate update to those trainable parameters.

Using Adam Manually

The same idea works with Adam.

import tensorflow as tf

x = tf.Variable(5.0)

optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.1
)

with tf.GradientTape() as tape:

    loss = (x - 2) ** 2

gradient = tape.gradient(loss, x)

optimizer.apply_gradients([
    (gradient, x)
])

print(x.numpy())

The optimizer is different, but the overall process is the same:

Calculate Loss
      ↓
Calculate Gradient
      ↓
Give Gradient to Optimizer
      ↓
Optimizer Updates Variable

Two Practical Examples

Example 1 — Start With Adam

optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)

model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

This is a sensible starting point for many beginner projects.

Example 2 — Try SGD

optimizer = tf.keras.optimizers.SGD(
    learning_rate=0.01,
    momentum=0.9
)

model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

You can then compare the validation results against the Adam version.

Common Mistake

A common mistake is changing everything at once:

Change:
→ Optimizer
→ Learning rate
→ Batch size
→ Number of layers
→ Number of neurons
→ Activation functions

Then the model improves and you have no idea what caused the improvement.

A better experiment is:

Keep model the same

Change:
Optimizer only

Compare results

Then change another parameter separately.

Remember This

Optimizer
    ↓
Controls weight updates


Adam
    ↓
tf.keras.optimizers.Adam()


SGD
    ↓
tf.keras.optimizers.SGD()


SGD + Momentum
    ↓
tf.keras.optimizers.SGD(
    momentum=0.9
)

The optimizer is normally passed to:

model.compile()

The complete training flow is:

model.fit()
    ↓
Forward Pass
    ↓
Loss
    ↓
Backpropagation
    ↓
Gradients
    ↓
Optimizer
    ↓
Weights Updated
    ↓
Repeat

The key Python pattern to remember is:

optimizer = tf.keras.optimizers.Adam(
    learning_rate=0.001
)

model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

model.fit(
    X_train,
    y_train,
    epochs=10
)