Choosing an Activation Function

The Three Activation Functions

In this lesson we have learned three important activation functions:

A Simple Way to Choose

Start by asking what your output represents.

Binary probability?
        ↓
     Sigmoid


Hidden layer?
        ↓
       ReLU


Need values from -1 to 1?
        ↓
       Tanh

This is a useful starting point, but it is not an absolute rule. The architecture and training problem still matter.

When Should You Use Sigmoid?

Sigmoid produces a value between 0 and 1.

Sigmoid → 0 to 1

This makes it useful when the output represents a probability for a binary decision.

Example — Spam Detection

Suppose we want to predict whether an email is spam.

0 = Not Spam
1 = Spam

The model might produce:

0.92

We can interpret this as approximately a 92% predicted probability of the positive class, depending on how the model is trained and calibrated.

Another Sigmoid Example

Imagine predicting whether a customer will purchase a product.

Output = 0.15

The model is predicting a relatively low probability for the positive class.

So Sigmoid is useful when you need a single probability between 0 and 1 for a binary output.

When Should You Use ReLU?

ReLU is commonly used in hidden layers of neural networks.

ReLU(x) = max(0, x)

It keeps positive values and converts negative values to zero.

For example:

ReLU(-5) = 0
ReLU( 3) = 3

ReLU is popular because it is simple to calculate and provides a useful gradient for positive inputs.

Example — Image Recognition

Suppose a neural network is learning to recognize images of cats and dogs.

The hidden layers need to learn patterns such as edges, shapes, textures, and more complex features.

ReLU is commonly used in these hidden layers.

When Should You Use Tanh?

Tanh produces values between -1 and 1.

Tanh → -1 to 1

This makes it useful when negative, zero, and positive values are meaningful.

For example:

-1 = Negative
 0 = Neutral
+1 = Positive

Example — Representing Positive and Negative Information

Imagine a model needs an internal value where negative means one direction, positive means the opposite direction, and zero means neutral.

Tanh naturally provides this zero-centered range.

Sigmoid vs Tanh vs ReLU

Common Starting Choices

Example 1 — Binary Classification

Suppose we are building a model that predicts whether a customer will cancel a subscription.

Cancel
  ↓
Yes or No

This is a binary classification problem.

If we want one output representing the probability of the positive class, Sigmoid is a common choice for the output layer.

Hidden Layers → ReLU

Output Layer → Sigmoid

Example 2 — Image Classification

Suppose we want a neural network to recognize objects in images.

Image
  ↓
Hidden Layers
  ↓
ReLU
  ↓
More Hidden Layers
  ↓
Output

ReLU is a common starting choice for many hidden layers. The final output activation depends on the exact classification problem.

Example 3 — Sequence Models

Tanh has historically been important in recurrent neural networks because it naturally represents values between -1 and 1.

Sequence
   ↓
RNN
   ↓
Tanh
   ↓
Hidden State

You will understand this more deeply when we reach the RNN and LSTM lessons.

Don't Choose an Activation Function Blindly

A common beginner mistake is thinking:

"ReLU is popular, so I should use ReLU everywhere."

That is wrong.

The activation function depends on the layer and the problem. The output layer especially needs to match the type of prediction you are making.

For example:

Binary probability
        ↓
     Sigmoid


Many hidden layers
        ↓
       ReLU


Some zero-centered representations
        ↓
       Tanh

Simple Decision Guide

What does the output need to represent?
                │
                ├── Binary probability
                │       ↓
                │    Sigmoid
                │
                ├── Hidden layer
                │       ↓
                │     ReLU
                │
                └── Value centered around zero
                        ↓
                       Tanh

The Big Picture

The hidden layers and output layer do not necessarily need the same activation function.