Stochastic Gradient Descent

What Is Stochastic Gradient Descent?

In basic Gradient Descent, the model uses the entire training dataset to calculate a gradient before updating the weights.

Stochastic Gradient Descent does something different: it takes one training example, calculates the gradient, and immediately updates the weights.

One Training Example
        ↓
Prediction
        ↓
Loss
        ↓
Gradient
        ↓
Update Weights
        ↓
Next Training Example
        ↓
Repeat

This makes SGD much more frequent in its weight updates than full-batch Gradient Descent.

Simple Example

Suppose our dataset contains four training examples:

Training Data:

Example 1
Example 2
Example 3
Example 4

Full Gradient Descent would first use all four examples:

Example 1 ─┐
Example 2 ─┤
Example 3 ─┼→ Calculate Gradient
Example 4 ─┘
                  ↓
             Update Weight

SGD instead processes one example and updates immediately:

Example 1
    ↓
Gradient
    ↓
Update Weight

Example 2
    ↓
Gradient
    ↓
Update Weight

Example 3
    ↓
Gradient
    ↓
Update Weight

Example 4
    ↓
Gradient
    ↓
Update Weight

Why Is It Called "Stochastic"?

"Stochastic" means involving randomness or probability.

In SGD, the training examples are commonly shuffled and processed individually, so each update is based on a different sample.

Dataset

1
2
3
4
5

       ↓ Shuffle

3
1
5
2
4

       ↓

Use one example
       ↓
Update

Use next example
       ↓
Update

...

The exact order can change from one training epoch to another.

SGD Update Formula

The basic update rule is the same idea as Gradient Descent:

new_weight =
    old_weight - learning_rate * gradient

The important difference is where the gradient comes from.

Gradient Descent:
Gradient calculated using the whole dataset


SGD:
Gradient calculated using one training example

Step-by-Step Numeric Example

Suppose our current weight is:

weight = 0.50

We process one training example and backpropagation gives:

gradient = 0.20

Our learning rate is:

learning_rate = 0.10

Apply the SGD update:

new_weight =
    0.50 - (0.10 × 0.20)

new_weight =
    0.50 - 0.02

new_weight =
    0.48

The weight changes immediately:

0.50
 ↓
0.48

Then SGD moves to the next training example.

Another Example

Suppose the next training example produces a different gradient:

Current weight = 0.48
Gradient       = 0.10
Learning rate  = 0.10

new_weight =
    0.48 - (0.10 × 0.10)

new_weight =
    0.48 - 0.01

new_weight =
    0.47

The weight is updated again:

0.50
 ↓
0.48
 ↓
0.47

Each training example can produce a different gradient, so each update can be different.

SGD With Four Training Examples

Imagine four examples produce these gradients:

Example 1 → gradient = 0.20
Example 2 → gradient = 0.10
Example 3 → gradient = -0.05
Example 4 → gradient = 0.08

Starting with:

weight = 0.50
learning_rate = 0.10

SGD processes them one at a time.

Example 1:

0.50 - (0.10 × 0.20)
= 0.48


Example 2:

0.48 - (0.10 × 0.10)
= 0.47


Example 3:

0.47 - (0.10 × -0.05)
= 0.475


Example 4:

0.475 - (0.10 × 0.08)
= 0.4675

Notice something important: the third gradient is negative, so the weight increases slightly.

0.50
 ↓
0.48
 ↓
0.47
 ↓
0.475
 ↓
0.4675

Why Does SGD Look Noisy?

Because each update is based on only one training example, the gradient can change significantly from one example to the next.

Example 1 → Gradient +0.30
Example 2 → Gradient +0.05
Example 3 → Gradient -0.10
Example 4 → Gradient +0.25
Example 5 → Gradient -0.02

Therefore, the weight updates may move back and forth.

Weight
  ↑
  |    ●
  |      ●
  |   ●
  |       ●
  |     ●
  |  ●
  +----------------→ Updates

This noise is not necessarily a problem. It can sometimes help SGD move through the loss landscape instead of making perfectly smooth but expensive updates.

What Is an Epoch in SGD?

An epoch means that the model has processed the entire training dataset once.

Suppose we have 4 examples:

Example 1
Example 2
Example 3
Example 4

SGD processes all four:

Example 1 → Update
Example 2 → Update
Example 3 → Update
Example 4 → Update

        ↓

One Epoch Completed

If we train for 3 epochs, the dataset is processed three times.

Epoch 1
Example 1 → Update
Example 2 → Update
Example 3 → Update
Example 4 → Update

Epoch 2
Example 1 → Update
Example 2 → Update
Example 3 → Update
Example 4 → Update

Epoch 3
Example 1 → Update
Example 2 → Update
Example 3 → Update
Example 4 → Update

SGD and Batch Size

Strictly speaking, pure SGD uses a batch size of 1.

Batch Size = 1

Example 1
   ↓
Update

Example 2
   ↓
Update

Example 3
   ↓
Update

This is different from mini-batch training, where multiple examples are used for each update.

SGD:

1 example
   ↓
1 update


Mini-Batch:

32 examples
   ↓
1 update

Mini-batch gradient descent will be covered separately.

Stochastic Gradient Descent With Python

Here is a simplified example showing the core SGD idea.

weight = 0.50
learning_rate = 0.10

gradients = [0.20, 0.10, -0.05, 0.08]

for gradient in gradients:

    weight = (
        weight
        - learning_rate * gradient
    )

    print(weight)

Output:

0.48
0.47
0.475
0.4675

Each value in gradients represents the gradient calculated from one training example.

Simple SGD Function

We can separate the update logic into a function:

def sgd_update(weight, gradient, learning_rate):

    return weight - learning_rate * gradient


weight = 0.50
learning_rate = 0.10

gradients = [0.20, 0.10, -0.05, 0.08]

for gradient in gradients:

    weight = sgd_update(
        weight,
        gradient,
        learning_rate
    )

    print(weight)

Output:

0.48
0.47
0.475
0.4675

SGD Training Flow

Training Dataset
      ↓
Shuffle Dataset
      ↓
Take One Example
      ↓
Forward Propagation
      ↓
Calculate Loss
      ↓
Backpropagation
      ↓
Calculate Gradient
      ↓
Update Weights
      ↓
Take Next Example
      ↓
Repeat
      ↓
All Examples Processed
      ↓
Epoch Completed

Then another epoch begins, usually with the data shuffled again.

Gradient Descent vs SGD

Full Gradient Descent

Entire Dataset
      ↓
Calculate Gradient
      ↓
One Weight Update


SGD

One Example
      ↓
Calculate Gradient
      ↓
One Weight Update
      ↓
Next Example
      ↓
Calculate Gradient
      ↓
Another Weight Update

So the key difference is the amount of data used to calculate each update.

Gradient Descent
→ Entire dataset per update


SGD
→ One example per update

Why Use SGD?

One major advantage is that each update is computationally smaller because it uses only one training example.

It can also begin updating the model immediately instead of waiting to process the entire dataset.

Large Dataset

Full Gradient Descent
→ Process everything
→ Calculate update


SGD
→ Process one example
→ Update immediately
→ Process next example
→ Update immediately

What Is the Disadvantage?

The updates can be noisy because one training example may not represent the entire dataset.

Example 1
→ Move this way

Example 2
→ Move another way

Example 3
→ Move back

Example 4
→ Move forward again

As a result, the loss may fluctuate instead of decreasing smoothly at every update.

This is one reason modern deep-learning systems commonly use mini-batches rather than pure batch Gradient Descent or pure single-example SGD.

Easy Real-World Example

Imagine a teacher has 1,000 students and wants to improve a teaching method.

Full Gradient Descent is like asking all 1,000 students for feedback before making one change.

1,000 students
      ↓
Collect feedback
      ↓
Make one change

SGD is like asking one student, making a small adjustment, then asking another student.

Student 1
   ↓
Make adjustment

Student 2
   ↓
Make adjustment

Student 3
   ↓
Make adjustment

...

The second approach gives faster feedback, but individual feedback can be noisy or unrepresentative.

Where SGD Fits in Neural Network Training

Training Example
      ↓
Input Layer
      ↓
Neural Network
      ↓
Prediction
      ↓
Loss Function
      ↓
Loss
      ↓
Backpropagation
      ↓
Gradient
      ↓
SGD
      ↓
Updated Weights
      ↓
Next Training Example
      ↓
Repeat

Remember the separation of responsibilities:

Loss Function
→ Measures the error


Backpropagation
→ Calculates gradients


SGD
→ Uses those gradients
  to update weights

One Important Point

Don't confuse "SGD" with "any optimizer that uses a batch." In strict terminology, SGD means the gradient is estimated from a single training example.

In modern machine-learning libraries, however, the term SGD is often used for an optimizer that operates on mini-batches as well. The underlying optimizer is still called SGD, while the training process may use a batch size greater than 1.

For this course, remember the simple distinction first:

Pure SGD
Batch Size = 1

Mini-Batch Gradient Descent
Batch Size > 1

Remember This

Stochastic Gradient Descent

1. Shuffle the training data
2. Take one training example
3. Make a prediction
4. Calculate the loss
5. Calculate the gradient
6. Update the weights
7. Take the next example
8. Repeat


Basic formula:

new_weight =
    old_weight
    - learning_rate × gradient

The most important sentence is: SGD updates the model's weights using the gradient calculated from one training example at a time.