Categorical Cross-Entropy

What Is Multi-Class Classification?

Multi-class classification means that a model must choose one class from more than two possible classes.

Cat
Dog
Horse

Or:

Apple
Banana
Orange

Or:

Car
Truck
Motorcycle
Bus

Unlike binary classification, there are more than two possible classes.

Binary Cross-Entropy vs Categorical Cross-Entropy

Binary Classification

Two classes

Example:
Spam / Not Spam

        ↓

Binary Cross-Entropy


Multi-Class Classification

Multiple classes

Example:
Cat / Dog / Horse

        ↓

Categorical Cross-Entropy

This distinction is important. Do not use BCE and categorical cross-entropy interchangeably just because both use the word "cross-entropy".

What Does the Model Predict?

In a typical multi-class classification problem, the model produces a probability for each class.

Suppose the model needs to classify an image as one of these three classes:

Cat
Dog
Horse

The model might produce:

Cat   = 0.10
Dog   = 0.80
Horse = 0.10

These probabilities add up to 1:

0.10 + 0.80 + 0.10 = 1.00

The model is therefore saying:

Cat   → 10%
Dog   → 80%
Horse → 10%

Compare the Prediction With the Actual Class

Suppose the image is actually a dog.

Actual Class = Dog

The model predicted:

Cat   = 0.10
Dog   = 0.80
Horse = 0.10

The important probability is the probability assigned to the correct class:

Correct Class = Dog
Probability   = 0.80

Because the model gave the correct class a high probability, the loss is relatively small.

What If the Model Is Wrong?

Suppose the image is still actually a dog.

Actual Class = Dog

But the model predicts:

Cat   = 0.90
Dog   = 0.05
Horse = 0.05

The correct class is Dog, but the model gave Dog only 5% probability.

Correct Class = Dog
Probability   = 0.05
        ↓
Very low probability
        ↓
Large loss

This is the central idea of categorical cross-entropy.

Why Does Confidence Matter?

Consider three predictions for an image that is actually a dog.

Prediction A

Cat   = 0.10
Dog   = 0.80
Horse = 0.10


Prediction B

Cat   = 0.20
Dog   = 0.50
Horse = 0.30


Prediction C

Cat   = 0.90
Dog   = 0.05
Horse = 0.05

Look only at the probability assigned to the correct class:

Prediction A → Dog = 0.80
Prediction B → Dog = 0.50
Prediction C → Dog = 0.05

Therefore:

Dog probability = 0.80
        ↓
Small loss


Dog probability = 0.50
        ↓
Larger loss


Dog probability = 0.05
        ↓
Very large loss

How Is the Correct Class Represented?

One common representation is called one-hot encoding.

Suppose we have three classes:

Cat
Dog
Horse

We can represent them as:

Cat   → [1, 0, 0]
Dog   → [0, 1, 0]
Horse → [0, 0, 1]

If the actual image is a dog:

Actual = [0, 1, 0]

The model might produce:

Prediction = [0.10, 0.80, 0.10]

The second position corresponds to Dog, so the loss focuses on the probability assigned to that correct class.

Categorical Cross-Entropy Formula

The formula for one example is:

Loss = -Σ yᵢ log(pᵢ)

The symbols mean:

yᵢ
↓
Actual value for class i

pᵢ
↓
Predicted probability for class i

Σ
↓
Sum across all classes

log
↓
Natural logarithm

With one-hot encoded targets, only the correct class has a target value of 1, so the calculation effectively focuses on the predicted probability of that class.

Calculate Categorical Cross-Entropy

Suppose the actual class is Dog:

Actual = [0, 1, 0]

The model predicts:

Prediction = [0.10, 0.80, 0.10]

The formula is:

Loss = -Σ yᵢ log(pᵢ)

Substitute the values:

Loss =
-[
    0 × log(0.10)
  + 1 × log(0.80)
  + 0 × log(0.10)
]

The terms multiplied by zero disappear:

Loss = -log(0.80)

Therefore:

Loss ≈ 0.223

This is relatively small because the model assigned 80% probability to the correct class.

Example of a Bad Prediction

The actual class is still Dog:

Actual = [0, 1, 0]

But the model predicts:

Prediction = [0.90, 0.05, 0.05]

The correct class, Dog, received only 0.05 probability.

Therefore:

Loss = -log(0.05)

Loss ≈ 2.996

Compare the two predictions:

Correct class probability = 0.80
Loss ≈ 0.223


Correct class probability = 0.05
Loss ≈ 2.996

The model that assigned very little probability to the correct class receives a much larger loss.

Softmax and Categorical Cross-Entropy

In a typical multi-class neural-network classifier, Softmax is used at the output layer to convert raw model outputs into probabilities that sum to 1.

Neural Network
      ↓
Raw Outputs
      ↓
Softmax
      ↓
Class Probabilities
      ↓
Categorical Cross-Entropy
      ↓
Loss

For example:

Raw Outputs

Cat   = 1.2
Dog   = 3.0
Horse = 0.8

        ↓

Softmax

        ↓

Cat   = 0.10
Dog   = 0.80
Horse = 0.10

The probabilities are then compared with the actual class using categorical cross-entropy.

Real-World Example — Animal Classification

Imagine an image classification model that identifies animals.

Classes:

Cat
Dog
Horse
Elephant

An image is actually a horse.

Actual = Horse

The model predicts:

Cat      = 0.05
Dog      = 0.10
Horse    = 0.80
Elephant = 0.05

The model gave the correct class 80% probability. Therefore, the loss is relatively small.

If the model instead predicts:

Cat      = 0.70
Dog      = 0.20
Horse    = 0.05
Elephant = 0.05

The correct class receives only 5%, so the loss becomes much larger.

Calculate Categorical Cross-Entropy With Python

We can calculate a simple one-hot encoded example manually using Python.

import math

actual = [0, 1, 0]

prediction = [0.10, 0.80, 0.10]

loss = 0

for y, p in zip(actual, prediction):

    loss += y * math.log(p)

loss = -loss

print("Categorical Cross-Entropy:", loss)

The result is approximately:

Categorical Cross-Entropy: 0.223

Understand the Python Code

1. Import math

import math

We need the logarithm function.

2. Store the actual class

actual = [0, 1, 0]

The second position represents Dog, so Dog is the correct class.

3. Store predictions

prediction = [0.10, 0.80, 0.10]

The model assigns Dog an 80% probability.

4. Start the loss at zero

loss = 0

5. Process each class

for y, p in zip(actual, prediction):

    loss += y * math.log(p)

The actual values and predicted probabilities are processed together.

Cat:
y = 0
p = 0.10


Dog:
y = 1
p = 0.80


Horse:
y = 0
p = 0.10

Because only Dog has an actual value of 1, only its predicted probability contributes to this one-hot cross-entropy calculation.

6. Make the loss positive

loss = -loss

The logarithm of probabilities between 0 and 1 is negative, so the negative sign makes the loss positive.

Compare Different Predictions

Suppose the actual class is Dog.

Actual = Dog

Compare these predictions:

Model A

Cat   = 0.05
Dog   = 0.90
Horse = 0.05


Model B

Cat   = 0.20
Dog   = 0.50
Horse = 0.30


Model C

Cat   = 0.80
Dog   = 0.10
Horse = 0.10

Look at only the probability assigned to the correct class:

Model A → Dog = 0.90
Model B → Dog = 0.50
Model C → Dog = 0.10

Therefore, categorical cross-entropy will generally rank their losses like this:

Model A → Smallest Loss
Model B → Larger Loss
Model C → Largest Loss

What Does Categorical Cross-Entropy Care About?

It cares strongly about the probability assigned to the correct class.

Correct class probability
          ↓
Higher probability
          ↓
Smaller loss


Correct class probability
          ↓
Lower probability
          ↓
Larger loss

This means a model is not only judged by whether its highest-probability class is correct. Its probability distribution also matters.

Prediction vs Loss

Suppose the model outputs:

Cat   = 0.40
Dog   = 0.50
Horse = 0.10

The predicted class is Dog because Dog has the highest probability.

But the loss also considers how much probability was assigned to the actual class.

Predicted Class
      ↓
Highest Probability


Categorical Cross-Entropy
      ↓
Probability Assigned
to the Correct Class

How Does It Help the Neural Network Learn?

Input Image
     ↓
Neural Network
     ↓
Softmax
     ↓
Class Probabilities
     ↓
Categorical Cross-Entropy
     ↓
Loss
     ↓
Backpropagation
     ↓
Optimizer
     ↓
Updated Weights
     ↓
Repeat

During training, the network adjusts its weights so that it can assign higher probabilities to the correct classes.

Binary vs Multi-Class Example

Binary Classification

Is this email spam?

Not Spam
Spam

        ↓

Binary Cross-Entropy


Multi-Class Classification

What animal is this?

Cat
Dog
Horse

        ↓

Categorical Cross-Entropy

The number of possible classes is one of the key differences.

Categorical Cross-Entropy in Keras

In a real deep-learning project, a framework handles the loss calculation for us.

For one-hot encoded labels, Keras can use:

model.compile(
    optimizer="adam",
    loss="categorical_crossentropy"
)

The model can then calculate categorical cross-entropy during training.

What If We Don't Use One-Hot Encoding?

There is another common setup called Sparse Categorical Cross-Entropy.

Instead of representing Dog as:

[0, 1, 0]

we can represent it with a class index:

Dog = 1

In Keras:

model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy"
)

Both approaches are used in multi-class classification. The correct choice depends on how your target labels are represented.

Categorical Cross-Entropy in One Picture

Input
  ↓
Neural Network
  ↓
Softmax
  ↓
Class Probabilities

Cat   = 0.10
Dog   = 0.80
Horse = 0.10

  ↓

Actual Class = Dog

  ↓

Correct Class Probability = 0.80

  ↓

Categorical Cross-Entropy

  ↓

Loss ≈ 0.223