Calculate Loss With Python

Basic Loss Calculation

Every loss calculation starts with two important things:

Actual Value
     +
Predicted Value
     ↓
Loss Function
     ↓
Loss Value

For example:

Actual     = 100
Prediction = 90

The prediction is different from the actual value, so there is some error.

Start With a Simple Error

Before using a real loss function, let's calculate the simple difference between the actual value and the prediction.

actual = 100
prediction = 90

error = actual - prediction

print(error)

Output:

10

The prediction is 10 away from the actual value.

But this is not yet a proper loss function. Different loss functions transform prediction errors in different ways.

Calculate Mean Squared Error

MSE is commonly used for regression problems.

The formula is:

MSE = average((actual - prediction)²)

Let's calculate it with Python.

actual = [10, 20, 30]
prediction = [12, 18, 29]

errors = []

for y, p in zip(actual, prediction):

    error = y - p
    squared_error = error ** 2

    errors.append(squared_error)

mse = sum(errors) / len(errors)

print("MSE:", mse)

Output:

MSE: 3.0

Understand the MSE Calculation

Our actual values are:

Actual:

[10, 20, 30]

Our predictions are:

Prediction:

[12, 18, 29]

Calculate each error:

10 - 12 = -2
20 - 18 =  2
30 - 29 =  1

Square each error:

(-2)² = 4
( 2)² = 4
( 1)² = 1

Now calculate the average:

MSE = (4 + 4 + 1) / 3

MSE = 9 / 3

MSE = 3

That's why Python produced:

MSE = 3.0

Create an MSE Function

Instead of writing the calculation every time, we can create a reusable Python function.

def mean_squared_error(actual, prediction):

    errors = []

    for y, p in zip(actual, prediction):

        error = y - p
        squared_error = error ** 2

        errors.append(squared_error)

    return sum(errors) / len(errors)


actual = [10, 20, 30]
prediction = [12, 18, 29]

loss = mean_squared_error(actual, prediction)

print("MSE:", loss)

Output:

MSE: 3.0

Now we can reuse the same function with different predictions.

What Happens When Predictions Improve?

First:

actual = [10, 20, 30]

prediction = [12, 18, 29]

MSE = 3.0

Now make the predictions closer to the actual values:

actual = [10, 20, 30]

prediction = [10, 21, 30]

MSE = 0.333...

The loss became smaller because the predictions became more accurate.

Prediction Error ↓
        ↓
MSE ↓

Calculate Binary Cross-Entropy

Binary Cross-Entropy is commonly used for binary classification.

For example:

0 = Not Spam
1 = Spam

Suppose the actual answer is:

Actual = 1

And the model predicts:

Prediction = 0.9

The BCE formula is:

Loss = -[y log(p) + (1-y) log(1-p)]

We can calculate this using Python.

import math

actual = 1
prediction = 0.9

loss = -(
    actual * math.log(prediction)
    + (1 - actual) * math.log(1 - prediction)
)

print("BCE:", loss)

Output:

BCE: 0.10536051565782628

Understand the BCE Calculation

We have:

Actual = 1
Prediction = 0.9

Substitute these values into the formula:

Loss =
-[1 × log(0.9) + (1 - 1) × log(1 - 0.9)]

The second part becomes zero:

Loss = -log(0.9)

Python calculates:

-math.log(0.9)

≈ 0.105

Because the model gave the correct class a high probability, the loss is relatively small.

What Happens With a Wrong Prediction?

Keep the actual answer as 1:

actual = 1

But change the prediction:

prediction = 0.1

Python:

import math

actual = 1
prediction = 0.1

loss = -(
    actual * math.log(prediction)
    + (1 - actual) * math.log(1 - prediction)
)

print("BCE:", loss)

Output:

BCE: 2.302585092994046

Compare:

Prediction = 0.9
BCE ≈ 0.105


Prediction = 0.1
BCE ≈ 2.303

The second model is confidently wrong, so it receives a much larger penalty.

Create a BCE Function

We can make BCE reusable:

import math


def binary_cross_entropy(actual, prediction):

    loss = -(
        actual * math.log(prediction)
        + (1 - actual) * math.log(1 - prediction)
    )

    return loss


print(binary_cross_entropy(1, 0.9))
print(binary_cross_entropy(1, 0.1))

Output:

0.10536051565782628
2.302585092994046

Calculate Categorical Cross-Entropy

Categorical Cross-Entropy is commonly used for multi-class classification.

Suppose we have three classes:

Cat
Dog
Horse

The actual class is Dog:

Actual = [0, 1, 0]

The model predicts:

Prediction = [0.10, 0.80, 0.10]

We can calculate the loss with Python:

import math

actual = [0, 1, 0]

prediction = [0.10, 0.80, 0.10]

loss = 0

for y, p in zip(actual, prediction):

    loss += y * math.log(p)

loss = -loss

print("Categorical Cross-Entropy:", loss)

Output:

Categorical Cross-Entropy: 0.2231435513142097

Understand the Categorical Calculation

The actual class is Dog:

Actual = [0, 1, 0]

The model predicts:

Prediction = [0.10, 0.80, 0.10]

The formula is:

Loss = -Σ yᵢ log(pᵢ)

Substitute the values:

Loss =
-[
    0 × log(0.10)
  + 1 × log(0.80)
  + 0 × log(0.10)
]

Only the correct class contributes:

Loss = -log(0.80)

Loss ≈ 0.223

The model assigned 80% probability to the correct class, so the loss is relatively small.

What Happens With a Bad Prediction?

The actual class is still Dog:

Actual = [0, 1, 0]

But now the model predicts:

Prediction = [0.90, 0.05, 0.05]

The correct class received only 5%.

Loss = -log(0.05)

Loss ≈ 2.996

Compare:

Correct class probability = 0.80
Loss ≈ 0.223


Correct class probability = 0.05
Loss ≈ 2.996

Again, lower probability for the correct class produces a larger loss.

Compare the Three Loss Calculations

Loss Function
      ↓
Depends on the problem


MSE
↓
Commonly used for regression


Binary Cross-Entropy
↓
Commonly used for binary classification


Categorical Cross-Entropy
↓
Commonly used for multi-class classification

The Python calculation changes because the mathematical definition of each loss function is different.

Complete Python Example

Here is a small example containing all three calculations.

import math


# -----------------------------
# MSE
# -----------------------------

actual_values = [10, 20, 30]
predicted_values = [12, 18, 29]

errors = []

for actual, prediction in zip(
    actual_values,
    predicted_values
):
    error = actual - prediction
    errors.append(error ** 2)

mse = sum(errors) / len(errors)

print("MSE:", mse)


# -----------------------------
# Binary Cross-Entropy
# -----------------------------

actual = 1
prediction = 0.9

bce = -(
    actual * math.log(prediction)
    + (1 - actual) * math.log(1 - prediction)
)

print("BCE:", bce)


# -----------------------------
# Categorical Cross-Entropy
# -----------------------------

actual = [0, 1, 0]
prediction = [0.10, 0.80, 0.10]

cce = 0

for y, p in zip(actual, prediction):

    cce += y * math.log(p)

cce = -cce

print("Categorical Cross-Entropy:", cce)

Example output:

MSE: 3.0

BCE: 0.10536051565782628

Categorical Cross-Entropy:
0.2231435513142097

Important: Probabilities Cannot Be Exactly 0 or 1

There is an important problem when calculating cross-entropy manually.

The logarithm of zero is undefined:

math.log(0)

→ Error

In practice, predictions are therefore usually clipped to a very small range away from exactly 0 and 1 when implementing the formula manually.

epsilon = 1e-15

prediction = max(
    min(prediction, 1 - epsilon),
    epsilon
)

This is one reason you should not replace a framework's production loss implementation with a simple educational formula.

How Real Deep Learning Code Does It

In real projects, you normally let TensorFlow/Keras or PyTorch calculate the loss.

For example, in Keras:

model.compile(
    optimizer="adam",
    loss="binary_crossentropy"
)

For a multi-class problem:

model.compile(
    optimizer="adam",
    loss="categorical_crossentropy"
)

The framework handles the mathematical details and provides optimized implementations.

Where Does Loss Fit Into Training?

Input Data
    ↓
Neural Network
    ↓
Prediction
    ↓
Loss Function
    ↓
Loss Value
    ↓
Backpropagation
    ↓
Calculate Gradients
    ↓
Update Weights
    ↓
Train Again

The important point is that calculating the loss is not the end of training. The loss is used by backpropagation to determine how the model's weights should change.

A Simple Training Example

Imagine a binary classifier starts with this prediction:

Actual = 1
Prediction = 0.20

BCE ≈ 1.609

After training updates the weights:

Actual = 1
Prediction = 0.70

BCE ≈ 0.357

After more training:

Actual = 1
Prediction = 0.90

BCE ≈ 0.105

The model is becoming more confident in the correct answer, so the BCE is decreasing.

Prediction

0.20
 ↓
0.70
 ↓
0.90

BCE

1.609
 ↓
0.357
 ↓
0.105