Why Do We Need LSTM?

Why Not Just Use a Basic RNN?

We already learned that an RNN can carry information from one time step to the next.

Input 1
   ↓
 RNN
   ↓
Hidden State
   ↓
Input 2
   ↓
 RNN
   ↓
Hidden State
   ↓
Input 3
   ↓
 RNN
   ↓
Output

This works well when the useful information is relatively close to the current input.

But a problem appears when the sequence becomes long.

Short-Term vs Long-Term Information

Imagine reading this sentence:

"John went to the store because
he needed to buy some milk."

When the model reaches "he", it needs to remember that "John" was the person being discussed.

This is a relatively short dependency.

John
 ↓
went
 ↓
to
 ↓
the
 ↓
store
 ↓
he

"he" → John

A basic RNN can often handle relationships like this.

What Happens With Long Sequences?

Now imagine a much longer sentence:

"John moved from India to the United States
when he was twenty years old. He studied
computer science, worked for several companies,
learned several programming languages, and after
many years of experience, he finally started
building machine learning systems.
What country did John originally come from?"

To answer the question, the model needs information from much earlier in the sequence:

John → India
        ↓
   many words
        ↓
Question
        ↓
Answer: India

This is a long-term dependency.

The important information appeared many time steps ago.

How Basic RNNs Carry Information

A basic RNN passes a hidden state from one time step to the next.

x₁
 ↓
RNN
 ↓
h₁
 ↓
RNN
 ↓
h₂
 ↓
RNN
 ↓
h₃
 ↓
RNN
 ↓
h₄
 ↓
...
 ↓
hₙ

The hidden state is supposed to carry useful information from previous time steps.

The problem is that information can become weaker as it passes through many steps.

The Vanishing Gradient Problem

During training, neural networks use backpropagation to calculate gradients and update their weights.

In a long RNN sequence, gradients can become extremely small as they are propagated backward through many time steps.

Large gradient
      ↓
  × 0.5
      ↓
  × 0.5
      ↓
  × 0.5
      ↓
  × 0.5
      ↓
Very small gradient

For example:

1.0
 ↓
0.5
 ↓
0.25
 ↓
0.125
 ↓
0.0625
 ↓
0.03125

After many repeated operations, the value can become extremely small.

When gradients become too small, the network has difficulty learning relationships between distant parts of a sequence.

Simple Example of the Problem

Suppose we have:

The animal didn't cross the road
because it was too tired.

The model needs to understand what "it" refers to.

animal
   ↓
didn't
   ↓
cross
   ↓
the
   ↓
road
   ↓
because
   ↓
it

The relationship is not necessarily difficult here because the sequence is short.

But imagine hundreds of words between "animal" and "it". Maintaining the important information becomes much harder for a basic RNN.

Another Example — Time Series

Consider a temperature sequence:

Day 1    → 20°C
Day 2    → 21°C
Day 3    → 22°C
...
Day 30   → 28°C

Suppose the temperature on Day 30 depends partly on a pattern that started many days earlier.

The model needs to preserve useful information over many time steps.

Important information
        ↓
Day 1
        ↓
Day 2
        ↓
Day 3
        ↓
...
        ↓
Day 30
        ↓
Prediction

This is where basic RNNs can struggle.

The Core Problem With Basic RNNs

Basic RNN

Previous information
        ↓
Hidden State
        ↓
Next time step
        ↓
Hidden State
        ↓
Next time step
        ↓
Hidden State
        ↓
...
        ↓
Current time step

The hidden state has to carry information through many transformations.

Important information can gradually become less useful, especially across long sequences.

This makes learning long-term dependencies difficult.

So What Does LSTM Try to Solve?

LSTM stands for:

Long Short-Term Memory

LSTM was designed to provide a better mechanism for controlling which information should be kept, updated, or exposed.

Basic RNN

Information
     ↓
Hidden State
     ↓
Next Step


LSTM

Information
     ↓
Controlled Memory
     ↓
Next Step

The important idea is that LSTM gives the network a more controlled way to manage information over time.

A Simple Real-Life Example

Imagine you are studying for an exam.

You read hundreds of pages, but not every piece of information is important.

Information

Fact A → important
Fact B → not important
Fact C → important
Fact D → not important
Fact E → important

A good memory system should be able to decide:

"Keep this."

"Forget this."

"Update this."

"Use this information now."

That is the basic intuition behind the mechanisms that make LSTM useful.

We will study exactly how LSTM makes these decisions when we learn about LSTM Gates.

Basic RNN vs LSTM

Basic RNN

Sequence
   ↓
Hidden State
   ↓
Next Step
   ↓
Hidden State
   ↓
Next Step


LSTM

Sequence
   ↓
Memory + Hidden State
   ↓
Controlled Information Flow
   ↓
Next Step
   ↓
Memory + Hidden State

The important difference is not simply that "LSTM has more memory."

More accurately, LSTM introduces mechanisms that help control information flow and preserve useful information over longer sequences.

When Is LSTM Useful?

LSTM can be useful when relationships between data points occur over longer periods of a sequence.

Text

Earlier word
    ↓
Many words
    ↓
Current word

Time Series

Earlier measurement
    ↓
Many time steps
    ↓
Current prediction

Common sequence problems include text, speech, sensor data, and time-series forecasting.

LSTM Does Not "Remember Everything"

This is an important distinction.

LSTM is not a perfect memory that stores every previous input forever.

Instead, it learns mechanisms for deciding what information should be retained, changed, or used.

Sequence
   ↓
Useful information?
   ↓
Keep / update / discard
   ↓
Continue processing

The exact mechanisms that perform these operations are called gates.

Why Do We Need Gates?

Suppose the sequence contains:

Old information
      +
New information

The network needs a way to determine what should happen to both.

Old information
       ↓
Should I keep it?
       ↓
    YES / NO


New information
       ↓
Should I store it?
       ↓
    YES / NO

LSTM gates provide learned mechanisms for this kind of information control.

We will study the actual gates in the next topic.

The Big Picture

Basic RNN
     ↓
Can process sequences
     ↓
Works well for shorter dependencies
     ↓
Problem with long-term dependencies
     ↓
Vanishing / exploding gradient difficulties
     ↓
Need better information management
     ↓
LSTM
     ↓
Controlled memory and information flow

Final Summary

The reason we need LSTM can be reduced to one problem:

Basic RNNs can struggle to learn
long-term dependencies.

The main reason is that information and gradients have to pass through many time steps.

Long Sequence
      ↓
Many RNN Steps
      ↓
Information becomes harder to preserve
      ↓
Gradients can become very small
      ↓
Learning long-term relationships becomes difficult

LSTM addresses this by introducing a more controlled memory mechanism.

LSTM

Remember useful information
          +
Forget unnecessary information
          +
Update information
          +
Produce useful output

In the next topics, we will see exactly how LSTM does this using its gates and cell state.