Mini-Batch Gradient Descent

What Is Mini-Batch Gradient Descent?

Mini-Batch Gradient Descent divides the training dataset into small groups called mini-batches.

The model processes one mini-batch, calculates the average gradient for that mini-batch, and then updates the weights.

Training Dataset
        ↓
Divide into Mini-Batches
        ↓
Mini-Batch 1
        ↓
Calculate Gradient
        ↓
Update Weights
        ↓
Mini-Batch 2
        ↓
Calculate Gradient
        ↓
Update Weights
        ↓
Repeat

It is essentially a compromise between full-batch Gradient Descent and SGD.

Simple Example

Suppose we have 12 training examples:

1  2  3  4  5  6
7  8  9  10 11 12

Suppose our batch size is 4.

We divide the dataset into three mini-batches:

Mini-Batch 1
[1, 2, 3, 4]

Mini-Batch 2
[5, 6, 7, 8]

Mini-Batch 3
[9, 10, 11, 12]

The model performs one weight update after processing each mini-batch.

[1, 2, 3, 4]
      ↓
Gradient
      ↓
Update Weights

[5, 6, 7, 8]
      ↓
Gradient
      ↓
Update Weights

[9, 10, 11, 12]
      ↓
Gradient
      ↓
Update Weights

What Is Batch Size?

The batch size tells us how many training examples are processed before the weights are updated.

For example:

Dataset size = 1000
Batch size   = 32

The model processes up to 32 examples, calculates the gradient, and updates the weights.

32 examples
     ↓
Gradient
     ↓
Weight Update

Then it processes the next 32 examples.

Common batch sizes include:

16
32
64
128
256

There is no single batch size that is always best. It depends on the dataset, model, hardware, and training problem.

Gradient Descent vs SGD vs Mini-Batch

Full Gradient Descent

1000 examples
     ↓
1 weight update


SGD

1 example
     ↓
1 weight update


Mini-Batch Gradient Descent

32 examples
     ↓
1 weight update

The difference is simply how many examples are used to calculate each update.

Full Batch
→ Batch Size = Entire Dataset


SGD
→ Batch Size = 1


Mini-Batch
→ Batch Size = Small Group

Step-by-Step Example

Suppose a mini-batch contains three training examples. Their gradients are:

Example 1 → 0.20
Example 2 → 0.10
Example 3 → 0.30

Mini-Batch Gradient Descent combines these gradients. A simple average gives:

Average Gradient =
    (0.20 + 0.10 + 0.30) / 3

Average Gradient =
    0.60 / 3

Average Gradient =
    0.20

Now suppose:

Weight        = 0.50
Learning Rate = 0.10
Gradient      = 0.20

Apply the update:

new_weight =
    0.50 - (0.10 × 0.20)

new_weight =
    0.48

So one mini-batch produced one weight update.

Why Do We Average the Gradients?

Each example can produce a different gradient. The mini-batch combines those individual gradients into one representative gradient for the batch.

Example 1 → Gradient
Example 2 → Gradient
Example 3 → Gradient
Example 4 → Gradient
       ↓
Combine Gradients
       ↓
Average Gradient
       ↓
Update Weights

This usually produces a more stable update than using only one training example.

Another Example

Suppose the gradients for one mini-batch are:

0.40
0.20
0.10
0.30

Calculate their average:

Average =
(0.40 + 0.20 + 0.10 + 0.30) / 4

Average =
1.00 / 4

Average =
0.25

If:

weight = 0.80
learning_rate = 0.10

Then:

new_weight =
0.80 - (0.10 × 0.25)

= 0.775

The weight changes from:

0.80
 ↓
0.775

Mini-Batches and Epochs

An epoch means that the entire training dataset has been processed once.

Suppose we have:

Dataset = 100 examples
Batch size = 20

The dataset is divided into:

Batch 1 → 20 examples
Batch 2 → 20 examples
Batch 3 → 20 examples
Batch 4 → 20 examples
Batch 5 → 20 examples

Therefore, one epoch contains five mini-batch updates.

Epoch 1

Batch 1 → Update
Batch 2 → Update
Batch 3 → Update
Batch 4 → Update
Batch 5 → Update

Epoch Completed

Mini-Batches and Iterations

One mini-batch normally produces one training iteration or optimizer update.

For example:

Dataset = 1000 examples
Batch size = 100

Number of updates per epoch:

1000 / 100 = 10

Therefore:

1 Epoch
= 10 Mini-Batches
= 10 Weight Updates

Mini-Batch Gradient Descent With Python

Here is a simplified example showing the basic idea.

weight = 0.50
learning_rate = 0.10

mini_batches = [
    [0.20, 0.10, 0.30],
    [0.05, 0.15, 0.10],
    [0.20, 0.25, 0.15]
]

for batch in mini_batches:

    average_gradient = sum(batch) / len(batch)

    weight = (
        weight
        - learning_rate * average_gradient
    )

    print(weight)

The important part is:

average_gradient =
    sum(batch) / len(batch)

The gradients inside the mini-batch are combined before the weight is updated.

Understanding the Python Code

weight = 0.50

This is our starting weight.

learning_rate = 0.10

This controls how large the weight update will be.

mini_batches = [
    [0.20, 0.10, 0.30],
    [0.05, 0.15, 0.10],
    [0.20, 0.25, 0.15]
]

Each inner list represents the gradients from one mini-batch.

for batch in mini_batches:

We process one mini-batch at a time.

average_gradient =
    sum(batch) / len(batch)

We calculate the average gradient for the current batch.

weight =
    weight - learning_rate * average_gradient

We then update the weight using the Gradient Descent rule.

Complete Training Flow

Training Dataset
       ↓
Shuffle Dataset
       ↓
Split Into Mini-Batches
       ↓
Mini-Batch 1
       ↓
Forward Propagation
       ↓
Calculate Loss
       ↓
Backpropagation
       ↓
Calculate Gradients
       ↓
Average Gradients
       ↓
Update Weights
       ↓
Mini-Batch 2
       ↓
Repeat
       ↓
All Mini-Batches Completed
       ↓
Epoch Completed

This process is repeated for many epochs until the model learns useful parameters.

Why Use Mini-Batches?

Mini-batches provide a practical balance between full-batch Gradient Descent and SGD.

Full Batch

Very stable gradient
        +
Large computation


SGD

Very frequent updates
        +
Noisy gradient


Mini-Batch

Reasonably stable gradient
        +
Efficient updates

Mini-batches also work well with modern hardware because multiple examples can be processed together.

What Happens When Batch Size Changes?

Consider the same dataset with different batch sizes.

Dataset = 1000 examples


Batch Size = 1
→ 1000 updates per epoch


Batch Size = 10
→ 100 updates per epoch


Batch Size = 100
→ 10 updates per epoch


Batch Size = 1000
→ 1 update per epoch

Notice the trade-off: smaller batches produce more frequent updates, while larger batches produce fewer updates.

Small Batch vs Large Batch

Small Batch

Less memory
More frequent updates
More gradient noise


Large Batch

More memory
Fewer updates
More stable gradient estimate

Neither is automatically better. Choosing a batch size is a practical training decision.

Easy Real-World Example

Imagine a teacher has 100 students and wants feedback about a new teaching method.

Full Gradient Descent is like asking all 100 students before making one decision.

100 Students
      ↓
Collect All Feedback
      ↓
Make One Change

SGD is like asking one student at a time and changing the method after every student.

Student 1
 ↓
Change

Student 2
 ↓
Change

Student 3
 ↓
Change

Mini-Batch Gradient Descent is like asking 10 students, considering their combined feedback, and then making one change.

Students 1–10
      ↓
Combine Feedback
      ↓
Make One Change

Students 11–20
      ↓
Combine Feedback
      ↓
Make One Change

That is the basic idea of a mini-batch.

Do Not Confuse These Three

Gradient Descent

Uses entire dataset
for each update.


SGD

Uses one example
for each update.


Mini-Batch Gradient Descent

Uses a small group
for each update.

This distinction is one of the most important things to understand before moving to more advanced optimizers.

Remember This

Mini-Batch Gradient Descent

1. Shuffle the dataset
2. Divide data into small batches
3. Take one mini-batch
4. Make predictions
5. Calculate loss
6. Calculate gradients
7. Combine the gradients
8. Update weights
9. Take the next mini-batch
10. Repeat

The most important sentence is: Mini-Batch Gradient Descent uses a small group of training examples to calculate a gradient and update the model's weights.