Activation Functions With Python

What Will We Build?

We will create three Python functions:

sigmoid()
relu()
tanh()

Then we will give them the same input values and compare their outputs.

[-2, -1, 0, 1, 2]

1. Sigmoid With Python

Sigmoid converts an input into a value between 0 and 1.

The formula is:

sigmoid(x) = 1 / (1 + e^(-x))

Python implementation:

import math


def sigmoid(x):
    return 1 / (1 + math.exp(-x))

Now let's test it:

print(sigmoid(-2))
print(sigmoid(-1))
print(sigmoid(0))
print(sigmoid(1))
print(sigmoid(2))

The results are approximately:

0.119
0.269
0.500
0.731
0.881

Understand the Sigmoid Code

First we import Python's math module:

import math

Then we create a function:

def sigmoid(x):

The function receives a value called x.

Then we apply the formula:

return 1 / (1 + math.exp(-x))

For example:

sigmoid(0)

= 1 / (1 + e^0)

= 1 / 2

= 0.5

2. ReLU With Python

ReLU is much simpler.

ReLU(x) = max(0, x)

Python implementation:

def relu(x):
    return max(0, x)

Test it:

print(relu(-2))
print(relu(-1))
print(relu(0))
print(relu(1))
print(relu(2))

Output:

0
0
0
1
2

Understand the ReLU Code

The function receives x:

def relu(x):

Then Python compares zero and x:

return max(0, x)

Example:

relu(-5)

max(0, -5)

→ 0

And:

relu(5)

max(0, 5)

→ 5

3. Tanh With Python

Tanh converts values into a range between -1 and 1.

Python already provides Tanh through the math module.

def tanh(x):
    return math.tanh(x)

Test it:

print(tanh(-2))
print(tanh(-1))
print(tanh(0))
print(tanh(1))
print(tanh(2))

Output:

-0.964
-0.762
0.000
0.762
0.964

Understand the Tanh Code

We create a function:

def tanh(x):

Then Python's math module calculates Tanh:

return math.tanh(x)

For example:

tanh(-2)

→ -0.964

And:

tanh(2)

→ 0.964

Complete Python Code

Now let's put all three functions together.

import math


def sigmoid(x):
    return 1 / (1 + math.exp(-x))


def relu(x):
    return max(0, x)


def tanh(x):
    return math.tanh(x)


values = [-2, -1, 0, 1, 2]


for value in values:

    print("Input:", value)

    print("Sigmoid:", sigmoid(value))

    print("ReLU:", relu(value))

    print("Tanh:", tanh(value))

    print()

Understanding the Output

The important part is not memorizing every decimal. Look at how each activation function changes the input.

What Is Different?

Look at the input:

x = -2

Each function gives a different result:

Sigmoid(-2) ≈ 0.119

ReLU(-2) = 0

Tanh(-2) ≈ -0.964

Same input, but three different outputs.

Activation Function Inside a Neural Network

Remember that an activation function is normally applied after a neuron calculates its weighted sum and bias.

Inputs
   ↓
Weights
   ↓
Weighted Sum + Bias
   ↓
z
   ↓
Activation Function
   ↓
Output

For example:

z = 2

Sigmoid(z) → 0.881
ReLU(z)    → 2
Tanh(z)    → 0.964

Using NumPy

In machine learning, we often work with many values at once. NumPy makes this easy.

import numpy as np


values = np.array([-2, -1, 0, 1, 2])


sigmoid = 1 / (1 + np.exp(-values))

relu = np.maximum(0, values)

tanh = np.tanh(values)


print("Sigmoid:", sigmoid)
print("ReLU:", relu)
print("Tanh:", tanh)

The important idea is that NumPy can apply the calculation to the entire array.

Simple Neural Network Example

Imagine that a neuron calculates:

z = -1.5

Different activation functions produce:

Sigmoid(-1.5) ≈ 0.182

ReLU(-1.5) = 0

Tanh(-1.5) ≈ -0.905

The activation function determines how the neuron's calculated value is transformed before being passed forward.

Quick Guide