LSTM Gates

What Is a Gate?

A gate is a mechanism that helps the LSTM decide how much information should pass through.

Think of a gate like a door:

Closed door
    ↓
Information does not pass

Half open
    ↓
Some information passes

Fully open
    ↓
Most information passes

LSTM uses this idea with numbers between 0 and 1.

0 → block the information

0.5 → allow some information

1 → allow the information

These values are usually produced using the sigmoid activation function.

The Three LSTM Gates

LSTM
 │
 ├── Forget Gate
 │      ↓
 │   What should I forget?
 │
 ├── Input Gate
 │      ↓
 │   What new information should I store?
 │
 └── Output Gate
        ↓
     What should I output?

These gates work together to control the LSTM's memory and output.

A Simple Example

Imagine an LSTM reading this sentence:

"John lives in London.
He moved to Paris last year.
He now works as a developer."

Suppose the model needs to understand where John currently lives.

When it reads:

"He moved to Paris"

the old information:

John → London

may no longer be correct.

The LSTM needs to:

1. Forget old information
2. Store the new information
3. Produce the useful current information

This is exactly where the three gates become useful.

1. Forget Gate

The Forget Gate decides which information from the previous cell state should be kept and which should be removed.

Previous Memory
      ↓
Forget Gate
      ↓
What should I forget?

For example:

Old information:

John lives in London

New information:

John moved to Paris

The LSTM may decide that the old location London is no longer useful.

London
   ↓
Forget Gate
   ↓
Remove / reduce importance

The important point is that the gate does not simply make a human-like decision. It calculates a numerical value that controls how much of the old information remains.

How Does the Forget Gate Use Numbers?

Suppose the Forget Gate produces:

0.0 → completely forget
0.2 → mostly forget
0.5 → keep about half
0.8 → mostly keep
1.0 → keep

Imagine the old memory contains:

London → 1.0
Developer → 0.8
Tennis → 0.6

The Forget Gate might produce:

London → 0.1
Developer → 0.9
Tennis → 0.7

This means the LSTM is allowing very little of the old London information to remain while keeping much more of the other information.

2. Input Gate

After deciding what old information to forget, the LSTM needs to decide what new information should be stored.

Current Input
      ↓
Input Gate
      ↓
What new information should I store?

Continuing our example:

"John moved to Paris."

The LSTM can identify:

Paris
   ↓
Potentially useful new information
   ↓
Input Gate
   ↓
Store it in memory

So the Input Gate controls how much new information is allowed into the cell state.

Input Gate Using Numbers

Again, the gate produces values between 0 and 1.

0 → do not store
0.5 → store partially
1 → strongly store

For example:

Paris      → 0.95
Developer  → 0.20
Tennis     → 0.10

The LSTM is effectively giving high importance to the new information about Paris and lower importance to unrelated information.

3. Output Gate

The Output Gate decides what information from the current memory should be exposed as the LSTM's output.

Current Memory
      ↓
Output Gate
      ↓
What should I output?

For example, if the current task is:

"Where does John live?"

the useful output might be:

Paris

The Output Gate controls which information becomes part of the hidden state and is exposed to the next part of the network.

Output Gate Using Numbers

0 → do not expose
0.5 → partially expose
1 → strongly expose

The Output Gate therefore controls the current output, rather than directly deciding what remains in long-term memory.

How the Three Gates Work Together

Previous Cell State
        ↓
   ┌─────────────┐
   │ Forget Gate │
   └─────────────┘
        ↓
Remove unnecessary old information
        ↓
   ┌─────────────┐
   │  Input Gate │ ← Current Input
   └─────────────┘
        ↓
Add useful new information
        ↓
Updated Cell State
        ↓
   ┌─────────────┐
   │ Output Gate │
   └─────────────┘
        ↓
Current Hidden State / Output

This is the central idea of LSTM gates.

Easy Real-Life Analogy

Imagine your brain reading a news story.

New information arrives
        ↓
Should I forget old information?
        ↓
Should I remember this new information?
        ↓
What information should I use right now?

You can map these decisions to the LSTM gates:

Forget Gate
"What should I remove?"

Input Gate
"What should I remember?"

Output Gate
"What should I use as output?"

This analogy is useful for understanding the concept, but remember that an LSTM is performing numerical calculations, not literally thinking like a human.

Where Do the Gates Work?

The gates mainly control information flowing through the LSTM's cell state and hidden state.

                 Current Input
                      ↓
Previous State → [  LSTM  ] → New State
                      │
                ┌─────┴─────┐
                ↓           ↓
           Cell State   Hidden State
              Memory        Output

The Forget and Input Gates help determine how the cell state changes.

The Output Gate determines how the current internal information is exposed through the hidden state.

Complete Example

Let's process these two sentences:

"John lives in London.
John moved to Paris."

Step 1 — The LSTM remembers:

John → London

Step 2 — It receives:

John moved to Paris

Step 3 — Forget Gate:

Old location: London
        ↓
Forget Gate
        ↓
Reduce old location information

Step 4 — Input Gate:

New location: Paris
        ↓
Input Gate
        ↓
Store useful new information

Step 5 — Updated memory:

Current location → Paris

Step 6 — Output Gate:

Current useful information
        ↓
Output Gate
        ↓
Hidden State / Output

So when the model later needs the current location, the updated information can be used.

Why Do Gates Use Values Between 0 and 1?

LSTM gates commonly use the sigmoid activation function.

sigmoid(x) = 1 / (1 + e⁻ˣ)

The sigmoid function converts a value into a number between 0 and 1.

Input       Sigmoid Output

-5          ≈ 0.007
-2          ≈ 0.119
 0          = 0.500
 2          ≈ 0.881
 5          ≈ 0.993

This makes the result useful as a gate.

0
↓
Closed

0.5
↓
Partially open

1
↓
Open

Do Not Confuse the Gates

Forget Gate
     ↓
Controls OLD information


Input Gate
     ↓
Controls NEW information


Output Gate
     ↓
Controls CURRENT OUTPUT

A simple way to remember them is:

Forget → Remove old information

Input → Add new information

Output → Produce useful information

Complete LSTM Information Flow

Previous Cell State
        +
Previous Hidden State
        +
Current Input
        │
        ↓
┌───────────────────────┐
│      Forget Gate      │
│   Remove old info     │
└───────────────────────┘
        ↓
┌───────────────────────┐
│       Input Gate      │
│    Add new info       │
└───────────────────────┘
        ↓
   Updated Cell State
        ↓
┌───────────────────────┐
│      Output Gate      │
│   Select output info  │
└───────────────────────┘
        ↓
   Hidden State
        ↓
   Next Time Step

The Big Picture

You do not need to memorize the mathematical equations yet. First understand the decision each gate makes.

              LSTM
                │
       ┌────────┼────────┐
       ↓        ↓        ↓
    Forget     Input    Output
       │        │        │
       ↓        ↓        ↓
    Remove     Add     Expose
     old       new     current
    info       info     info

Once this idea is clear, the mathematical formulas become much easier to understand.

Final Summary

Forget Gate
→ Decides what old information to keep or forget.

Input Gate
→ Decides what new information to add.

Output Gate
→ Decides what information to expose as output.

Together, these gates give LSTM control over information flowing through its memory.

OLD MEMORY
    ↓
Forget
    ↓
Remove unnecessary information
    ↓
Add useful new information
    ↓
NEW MEMORY
    ↓
Output useful information

This controlled information flow is one of the main reasons LSTM can handle long-term dependencies better than a basic RNN.