Hidden State

What Is Hidden State?

An RNN processes sequence data one step at a time.

At each step, the RNN receives two things:

1. Current Input
2. Previous Hidden State

It then produces a new hidden state.

Current Input
      +
Previous Hidden State
      ↓
     RNN
      ↓
New Hidden State

The new hidden state is then passed to the next step.

Think of Hidden State as Memory

Imagine you are reading this sentence:

I → love → machine → learning

When you reach the word "learning", you do not forget the words that came before it.

You remember the context:

I
 ↓
love
 ↓
machine
 ↓
learning

An RNN works in a similar way. The hidden state carries useful information from earlier steps.

It is not human memory. It is a numerical representation learned by the neural network.

How Hidden State Works

Consider this simple sequence:

10 → 20 → 30

At the beginning, there is no previous information. We normally start with an initial hidden state.

Hidden State 0 = 0

Now the RNN processes the first value:

Input = 10
Previous Hidden State = 0

        10
         +
         0
         ↓
        RNN
         ↓
Hidden State 1

The RNN produces a new hidden state.

Now the second value arrives:

Input = 20
Previous Hidden State = Hidden State 1

        20
         +
 Hidden State 1
         ↓
        RNN
         ↓
Hidden State 2

The third value works the same way:

Input = 30
Previous Hidden State = Hidden State 2

        30
         +
 Hidden State 2
         ↓
        RNN
         ↓
Hidden State 3

The Complete Flow

Input 1
   ↓
  RNN
   ↓
Hidden State 1
   ↓
   + Input 2
   ↓
  RNN
   ↓
Hidden State 2
   ↓
   + Input 3
   ↓
  RNN
   ↓
Hidden State 3

Notice that the hidden state keeps moving forward through the sequence.

That is what gives an RNN its ability to use information from previous steps.

Basic Hidden State Formula

A simplified RNN hidden-state equation is:

hₜ = tanh(Wₓₕ xₜ + Wₕₕ hₜ₋₁ + b)

Don't worry about memorizing the formula yet. Understand what each part represents.

xₜ
↓
Current input


hₜ₋₁
↓
Previous hidden state


Wₓₕ
↓
Weights applied to the current input


Wₕₕ
↓
Weights applied to the previous hidden state


b
↓
Bias


hₜ
↓
New hidden state

The important idea is:

Current Input
+
Previous Hidden State
↓
New Hidden State

Simple Numerical Example

To understand the idea, let's use a very simplified calculation.

Suppose:

Current input = 2
Previous hidden state = 3

Input weight = 0.5
Hidden-state weight = 0.2
Bias = 0

First calculate:

(2 × 0.5) + (3 × 0.2)

= 1 + 0.6

= 1.6

The RNN then applies an activation function such as tanh.

new hidden state = tanh(1.6)

The exact value is approximately:

0.9217

So the new hidden state becomes approximately 0.9217.

This is a simplified example. Real RNNs normally work with vectors containing many numbers rather than one number.

Hidden State With Python

We can demonstrate the basic idea using Python:

inputs = [10, 20, 30]

hidden_state = 0

for value in inputs:

    hidden_state = value + hidden_state

    print("Input:", value)
    print("Hidden State:", hidden_state)

Output:

Input: 10
Hidden State: 10

Input: 20
Hidden State: 30

Input: 30
Hidden State: 60

Here, we are not building a real RNN. We are only creating a simple example to understand the idea of carrying information forward.

Understand the Python Code

inputs = [10, 20, 30]

We create a sequence containing three values.

hidden_state = 0

We start with an initial hidden state of zero.

for value in inputs:

The loop processes each value in order.

hidden_state = value + hidden_state

This is the important line. The current value is combined with the previous hidden state.

10 + 0 = 10

20 + 10 = 30

30 + 30 = 60

Therefore, information from earlier steps affects later steps.

Hidden State in a Real RNN

In TensorFlow/Keras, we can create an RNN using:

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.SimpleRNN(4)
])

model.build(input_shape=(None, 5, 1))

model.summary()

The important part is:

tf.keras.layers.SimpleRNN(4)

This creates an RNN with 4 hidden units.

Instead of having one simple number as our demonstration, the real RNN uses a vector of values as its hidden state.

Hidden State

[
    0.12,
    0.83,
    -0.41,
    0.56
]

These values are learned representations of information from the sequence.

Why Is Hidden State Important?

Without a hidden state, the RNN would not have a mechanism for carrying information from previous steps.

Without Memory

Input 1 → RNN
Input 2 → RNN
Input 3 → RNN

With a hidden state:

Input 1 → RNN → Hidden State 1
                    ↓
Input 2 → RNN → Hidden State 2
                    ↓
Input 3 → RNN → Hidden State 3

The hidden state creates a connection between the steps.

Two Simple Examples

Example 1 — Sentence

The → movie → was → excellent

As the RNN processes each word, the hidden state carries information from the earlier words.

Example 2 — Temperature

25 → 27 → 29 → 31

The hidden state can carry information about the pattern seen in earlier temperature values.

Important: Hidden State Is Not the Input

Do not confuse the input with the hidden state.

Input
↓
The data currently being processed


Hidden State
↓
Information carried from previous steps

At each step, the RNN combines both.

Current Input
       +
Previous Hidden State
       ↓
      RNN
       ↓
New Hidden State

The Main Idea

Sequence
   ↓
Input 1
   ↓
RNN
   ↓
Hidden State 1
   ↓
Input 2 + Hidden State 1
   ↓
RNN
   ↓
Hidden State 2
   ↓
Input 3 + Hidden State 2
   ↓
RNN
   ↓
Hidden State 3

The hidden state is therefore the mechanism that allows an RNN to carry information through a sequence.

Final Summary

Hidden State
↓
RNN's carried information

At each step
↓
Current Input
+
Previous Hidden State
↓
RNN
↓
New Hidden State

Why?
↓
To carry information through the sequence

Simple idea
↓
Hidden State = Learned numerical representation
of information from previous steps.

The most important sentence to remember is:

The hidden state carries information
from previous steps to the current step.