What Is an RNN?

What Is an RNN?

RNN stands for Recurrent Neural Network.

It is a neural network designed for data where the order of the data matters.

For example, consider these words:

I → love → Python

The order matters because:

I love Python

is different from:

Python love I

An RNN processes the sequence step by step while carrying information from earlier steps.

The Basic Idea

A normal neural network can process an input and produce an output.

Input
  ↓
Neural Network
  ↓
Output

An RNN does something slightly different.

Current Input
      +
Previous Information
      ↓
     RNN
      ↓
New Information
      +
Output

The important part is the previous information.

This allows the RNN to use information from earlier steps when processing the current step.

Simple Example

Imagine an RNN processing this sentence:

I love deep learning

The RNN processes one word at a time.

Step 1

"I"
 ↓
RNN
 ↓
Hidden State


Step 2

"love"
 +
Previous Hidden State
 ↓
RNN
 ↓
New Hidden State


Step 3

"deep"
 +
Previous Hidden State
 ↓
RNN
 ↓
New Hidden State


Step 4

"learning"
 +
Previous Hidden State
 ↓
RNN
 ↓
New Hidden State

The hidden state carries information forward through the sequence.

Why Is This Important?

Suppose we process these words independently:

I
love
deep
learning

If every word is treated completely independently, the network does not naturally know what came before it.

An RNN instead passes information forward:

"I"
 ↓
Information about "I"
 ↓
"love"
 ↓
Information about "I love"
 ↓
"deep"
 ↓
Information about "I love deep"
 ↓
"learning"

This is the key idea behind an RNN.

What Is the Hidden State?

The hidden state is a set of numbers that represents information carried from previous steps.

You can think of it as the RNN's working state.

Input at time 1
      ↓
     RNN
      ↓
Hidden State 1
      ↓
Input at time 2
      ↓
     RNN
      ↓
Hidden State 2
      ↓
Input at time 3
      ↓
     RNN
      ↓
Hidden State 3

Notice that the previous hidden state is passed into the next step.

This is what gives an RNN its recurrent behavior.

What Happens Inside an RNN?

At every time step, the RNN receives two important things:

1. Current input
2. Previous hidden state

It then produces a new hidden state.

Current Input
      │
      ▼
   ┌───────┐
   │  RNN  │
   └───────┘
      ▲
      │
Previous Hidden State
      │
      ▼
New Hidden State

In many RNNs, the new hidden state can also be used to produce an output.

Current Input
      +
Previous Hidden State
      ↓
     RNN
      ↓
New Hidden State
      ↓
Output

The RNN Formula

The basic RNN hidden-state calculation can be written as:

hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁ + b)

This looks complicated, but the idea is simple.

xₜ
↓
Current Input


hₜ₋₁
↓
Previous Hidden State


Wₓ
↓
Weights for the current input


Wₕ
↓
Weights for previous information


b
↓
Bias


hₜ
↓
New Hidden State

In simple words:

New Hidden State
=
Current Input
+
Previous Hidden State
+
Learned Weights

The tanh function is an activation function used in the traditional basic RNN formulation.

Simple RNN With Python

Now let's create a very small RNN using TensorFlow and Keras.

import tensorflow as tf


model = tf.keras.Sequential([
    tf.keras.layers.SimpleRNN(
        16,
        input_shape=(5, 1)
    ),
    tf.keras.layers.Dense(1)
])


model.summary()

This creates a simple RNN with 16 hidden units.

Understanding the Python Code

import tensorflow as tf

This imports TensorFlow.

model = tf.keras.Sequential([

This creates a sequential model where layers are connected one after another.

tf.keras.layers.SimpleRNN(
    16,
    input_shape=(5, 1)
)

This creates the RNN layer.

The number 16 means the RNN has 16 units in its hidden state.

The input shape:

(5, 1)

means:

5 → Number of time steps
1 → Number of features at each time step

For example:

[
    [10],
    [20],
    [30],
    [40],
    [50]
]

contains 5 time steps and 1 feature at each step.

tf.keras.layers.Dense(1)

This adds one output neuron. It can be used, for example, when we want the RNN to produce one numerical prediction.

Example: Predict the Next Number

Suppose we have this sequence:

10 → 20 → 30 → 40 → 50

We could train an RNN to predict the next value.

Input:

10 → 20 → 30 → 40 → 50

        ↓
       RNN
        ↓

Prediction:

60

The RNN sees the sequence in order rather than treating the numbers as unrelated values.

Two Real-World Examples

Example 1 — Text

"I am learning"

I
 ↓
am
 ↓
learning

The meaning of the current word can depend on previous words.

Example 2 — Time Series

Monday    → 100
Tuesday   → 105
Wednesday → 110
Thursday  → 115
Friday    → ?

The previous observations can be used as part of the information for predicting the next value.

RNN vs Normal Neural Network

Normal Neural Network

Input
 ↓
Network
 ↓
Output


RNN

Input 1 ──→ RNN ──→ Hidden State 1
                    ↓
Input 2 ──→ RNN ──→ Hidden State 2
                    ↓
Input 3 ──→ RNN ──→ Hidden State 3
                    ↓
Input 4 ──→ RNN ──→ Output

The key difference is the flow of information from one time step to the next.

Why Is It Called "Recurrent"?

The word recurrent means something that happens repeatedly.

The same RNN cell is applied repeatedly across the sequence.

Input 1
   ↓
RNN Cell
   ↓
Input 2
   ↓
Same RNN Cell
   ↓
Input 3
   ↓
Same RNN Cell
   ↓
Input 4
   ↓
Same RNN Cell

The RNN does not normally create a completely new set of weights for every time step. The same learned weights are reused while the hidden state changes.

The Most Important Idea

If you remember only one thing from this lesson, remember this:

RNN = Current Input + Previous Hidden State

More visually:

                Previous Hidden State
                         │
                         ▼
Current Input ───────► RNN
                         │
                         ▼
                  New Hidden State
                         │
                         ▼
                  Next Time Step

The hidden state is what allows information to flow through the sequence.

A Simple Way to Think About It

Imagine you are reading a sentence one word at a time.

"The cat is sleeping"

When you reach the word "sleeping", you already have information about "The cat is".

An RNN works with a similar basic idea: information from earlier time steps is carried forward in its hidden state.

But be careful: the hidden state is not human memory. It is simply a learned numerical representation.

What an RNN Does Not Mean

An RNN does not mean that the model remembers everything perfectly.

It means that the model maintains a hidden state that is updated as it processes a sequence.

Previous Information
        ↓
Hidden State
        ↓
Updated with Current Input
        ↓
New Hidden State

Basic RNNs can struggle when important information occurs very far back in a long sequence. We will study those problems separately.

Final Summary

RNN
↓
Recurrent Neural Network

Used for
↓
Sequence Data

Examples
↓
Text
Time Series
Speech
Other Ordered Data

At Each Step
↓
Current Input
+
Previous Hidden State
↓
RNN
↓
New Hidden State

The Hidden State
↓
Carries information forward

The Main Idea
↓
Use previous information
while processing current input

So the simplest definition is:

An RNN is a neural network that processes
data step by step while carrying a hidden
state from previous steps to the next step.