Binary Cross-Entropy

First, What Is Binary Classification?

Binary classification means there are only two possible classes.

Spam / Not Spam

Pass / Fail

Cat / Not Cat

Fraud / Not Fraud

Disease / No Disease

We commonly represent the two classes using:

0 = Negative Class
1 = Positive Class

For example, for spam detection:

0 = Not Spam
1 = Spam

The Model Produces a Probability

For binary classification, the model often produces a value between 0 and 1.

0.0 → Very unlikely to be class 1
0.5 → Uncertain
1.0 → Very likely to be class 1

For example:

Prediction = 0.90

This means the model is giving a high probability to class 1.

If we are predicting spam:

Prediction = 0.90

→ Model thinks the email is very likely to be spam.

Example 1 — Correct Confident Prediction

Suppose the email is actually spam.

Actual = 1

The model predicts:

Prediction = 0.90

The model is highly confident and correct.

Actual
  1
  ↓
Prediction
  0.90
  ↓
Correct and confident
  ↓
Small BCE loss

Example 2 — Wrong Confident Prediction

Now suppose the email is still actually spam:

Actual = 1

But the model predicts:

Prediction = 0.01

The model is extremely confident that the email is not spam, even though it actually is spam.

Actual
  1
  ↓
Prediction
  0.01
  ↓
Very wrong and very confident
  ↓
Large BCE loss

This is one of the most important ideas behind Binary Cross-Entropy.

Why Does Confidence Matter?

Consider these three predictions when the actual answer is 1:

Actual = 1

Prediction A = 0.90
Prediction B = 0.60
Prediction C = 0.01

Prediction A is correct and confident.

Prediction B is correct but less confident.

Prediction C is extremely wrong and extremely confident.

0.90 → Small loss

0.60 → Larger loss

0.01 → Very large loss

BCE strongly penalizes predictions that are confidently wrong.

Binary Cross-Entropy Formula

The formula for one binary classification example is:

Loss = -[y log(p) + (1 - y) log(1 - p)]

The symbols mean:

y
↓
Actual answer

p
↓
Model's predicted probability

log
↓
Natural logarithm

Loss
↓
Binary Cross-Entropy value

The formula looks complicated at first, but it becomes much easier when we look at the two possible cases.

Case 1 — Actual Answer Is 1

If:

y = 1

Put 1 into the formula:

Loss = -[1 log(p) + (1 - 1) log(1 - p)]

The second part becomes zero:

Loss = -log(p)

So when the actual answer is 1, BCE mainly cares about how large the predicted probability p is.

Actual = 1

p close to 1
    ↓
Small loss

p close to 0
    ↓
Large loss

Case 2 — Actual Answer Is 0

If:

y = 0

The formula becomes:

Loss = -[0 log(p) + (1 - 0) log(1 - p)]

Therefore:

Loss = -log(1 - p)

Now BCE cares about how small the predicted probability for class 1 is.

Actual = 0

p close to 0
    ↓
Small loss

p close to 1
    ↓
Large loss

Calculate BCE Step by Step

Suppose:

Actual = 1
Prediction = 0.9

Because the actual answer is 1:

Loss = -log(0.9)

Approximately:

Loss ≈ 0.105

That is a relatively small loss because the prediction was close to the correct answer.

Calculate a Wrong Prediction

Now:

Actual = 1
Prediction = 0.1

The loss becomes:

Loss = -log(0.1)

Approximately:

Loss ≈ 2.303

Compare the two:

Actual = 1

Prediction = 0.9
BCE ≈ 0.105


Prediction = 0.1
BCE ≈ 2.303

The second prediction receives a much larger penalty.

Why Does BCE Use Logarithms?

You do not need to become an expert in logarithms to understand BCE.

The important behavior is:

Correct and confident
        ↓
Very small loss


Wrong and confident
        ↓
Very large loss

The logarithm creates a strong penalty when the model assigns a very small probability to the correct class.

For example:

-log(0.9)  ≈ 0.105
-log(0.5)  ≈ 0.693
-log(0.1)  ≈ 2.303
-log(0.01) ≈ 4.605

Notice how quickly the loss increases as the probability assigned to the correct class becomes very small.

BCE Handles Both Classes

BCE works for both possible actual values.

Actual = 1

Prediction = 0.9
→ Small loss


Actual = 0

Prediction = 0.1
→ Small loss

But:

Actual = 1

Prediction = 0.1
→ Large loss


Actual = 0

Prediction = 0.9
→ Large loss

The model is rewarded for assigning high probability to the correct class and penalized for assigning high probability to the wrong class.

Real-World Example — Spam Detection

Suppose we have an email classification model.

0 = Not Spam
1 = Spam

The email is actually spam:

Actual = 1

Model A predicts:

Prediction = 0.95

Model B predicts:

Prediction = 0.05

BCE gives a very different penalty:

Model A
0.95
↓
Correct and confident
↓
Very small loss


Model B
0.05
↓
Wrong and confident
↓
Very large loss

BCE vs MSE

Both MSE and BCE can measure prediction error, but they are commonly used for different kinds of problems.

MSE

Commonly used for:
Regression

Example:
Predict house price


BCE

Commonly used for:
Binary classification

Example:
Spam / Not Spam

The difference is not merely the formula. The loss function should match the type of prediction problem and output being modeled.

Sigmoid and Binary Cross-Entropy

In binary classification, a common neural-network setup uses a Sigmoid activation at the output.

Hidden Layers
     ↓
Output Neuron
     ↓
Sigmoid
     ↓
Probability
     ↓
Binary Cross-Entropy
     ↓
Loss

Sigmoid converts the output into a value between 0 and 1. BCE then evaluates that probability against the actual target.

For example:

Raw Output = 2.2

       ↓

Sigmoid

       ↓

Prediction ≈ 0.900

       ↓

BCE compares 0.900
with the actual answer.

Calculate BCE With Python

We can calculate Binary Cross-Entropy manually using Python.

import math

actual = 1
prediction = 0.9

loss = -(
    actual * math.log(prediction)
    + (1 - actual) * math.log(1 - prediction)
)

print("Binary Cross-Entropy:", loss)

The result is approximately:

Binary Cross-Entropy: 0.105

Understand the Python Code

1. Import math

import math

We need Python's logarithm function.

2. Store the actual answer

actual = 1

The correct class is 1.

3. Store the prediction

prediction = 0.9

The model gives class 1 a probability of 0.9.

4. Calculate BCE

loss = -(
    actual * math.log(prediction)
    + (1 - actual) * math.log(1 - prediction)
)

Because actual = 1, the expression reduces mathematically to:

loss = -math.log(0.9)

Which gives approximately:

0.105

Python Example With Both Classes

import math


def binary_cross_entropy(actual, prediction):

    loss = -(
        actual * math.log(prediction)
        + (1 - actual) * math.log(1 - prediction)
    )

    return loss


print(binary_cross_entropy(1, 0.9))
print(binary_cross_entropy(0, 0.1))

Both predictions are correct and confident, so both produce a relatively small loss.

Actual = 1
Prediction = 0.9

→ Small loss


Actual = 0
Prediction = 0.1

→ Small loss

Now Try Wrong Predictions

print(binary_cross_entropy(1, 0.1))
print(binary_cross_entropy(0, 0.9))

Both predictions are confidently wrong.

Actual = 1
Prediction = 0.1
→ Large loss


Actual = 0
Prediction = 0.9
→ Large loss

How BCE Helps Training

During training, the model repeatedly performs this process:

Input
  ↓
Neural Network
  ↓
Sigmoid
  ↓
Probability
  ↓
Binary Cross-Entropy
  ↓
Loss
  ↓
Backpropagation
  ↓
Update Weights
  ↓
Repeat

The goal is to adjust the model's parameters so that the predictions assign high probability to the correct class.

Binary Cross-Entropy in One Picture

Input
  ↓
Neural Network
  ↓
Sigmoid
  ↓
Prediction Probability
  ↓
Compare With Actual Answer
  ↓
Binary Cross-Entropy
  ↓
Loss Value
  ↓
Backpropagation
  ↓
Weight Updates