Positional Encoding

1. Why Do We Need Position?

Consider these two sentences:

Dog bites man.

Man bites dog.

Both sentences contain the same three words:

Dog
bites
man

But their meanings are completely different because the positions of the words changed.

Dog → position 1
bites → position 2
man → position 3

versus:

Man → position 1
bites → position 2
dog → position 3

Therefore, a Transformer needs information about where each token occurs.

2. Why Transformers Need Positional Encoding

RNNs naturally process data step by step:

Word 1
  ↓
Word 2
  ↓
Word 3
  ↓
Word 4

Because an RNN processes tokens sequentially, position is naturally part of the computation.

Transformers work differently.

Word 1 ─┐
Word 2 ─┤
Word 3 ─┼──→ Transformer
Word 4 ─┘

Tokens can be processed in parallel.

This is one of the major advantages of Transformers, but it creates a problem:

Where is Word 1?
Where is Word 2?
Where is Word 3?
Where is Word 4?

Positional encoding solves this problem.

3. What Is Positional Encoding?

Positional encoding is information added to token embeddings to tell the Transformer where each token is located in the sequence.

Token Embedding
       +
Positional Encoding
       ↓
Transformer Input

For example:

Token: "I"

Embedding
   +
Position information
   ↓
Final representation

The Transformer can now understand both:

WHAT the token is

and

WHERE the token is

4. Simple Example

Suppose our sentence is:

I love Python

Assign positions:

I       → 0
love    → 1
Python  → 2

Now suppose each word has an embedding:

I       → [0.2, 0.5, 0.1]
love    → [0.4, 0.3, 0.7]
Python  → [0.8, 0.1, 0.6]

We create a positional encoding for each position:

Position 0 → [0.0, 1.0, 0.0]
Position 1 → [0.84, 0.54, 0.01]
Position 2 → [0.91, -0.42, 0.02]

Then we add them together.

Token embedding
       +
Position encoding
       =
Transformer input

For the first word:

[0.2, 0.5, 0.1]
+
[0.0, 1.0, 0.0]
=
[0.2, 1.5, 0.1]

Now the representation contains both word information and position information.

5. How Positional Encoding Works

Sentence
    ↓
Tokenization
    ↓
Tokens
    ↓
Token Embeddings
    ↓
+
Positional Encoding
    ↓
Transformer
    ↓
Attention

The important part is:

Embedding + Position
        ↓
Information about
WHAT + WHERE

6. Sinusoidal Positional Encoding

The original Transformer architecture introduced a mathematical method using sine and cosine functions.

The formulas are:

PE(pos, 2i)
=
sin(
    pos / 10000^(2i / d_model)
)


PE(pos, 2i + 1)
=
cos(
    pos / 10000^(2i / d_model)
)

Do not worry about memorizing this formula yet.

The important idea is:

Position
   ↓
Sine / Cosine functions
   ↓
Position vector

Different positions produce different patterns.

7. Why Sine and Cosine?

Sine and cosine create smooth, repeating patterns.

Position 0 → pattern A
Position 1 → pattern B
Position 2 → pattern C
Position 3 → pattern D
...

Each position gets a unique numerical pattern.

The model can therefore distinguish:

Position 0
Position 1
Position 2
Position 3
...

Another useful property is that the encoding changes smoothly as positions change.

8. Positional Encoding With Python

Here is a simple implementation of sinusoidal positional encoding using TensorFlow.

import tensorflow as tf
import numpy as np


def positional_encoding(sequence_length, d_model):

    positions = np.arange(sequence_length)[:, np.newaxis]

    dimensions = np.arange(d_model)[np.newaxis, :]

    angle_rates = 1 / np.power(
        10000,
        (2 * (dimensions // 2)) / np.float32(d_model)
    )

    angle_rads = positions * angle_rates

    encoding = np.zeros(
        (sequence_length, d_model)
    )

    encoding[:, 0::2] = np.sin(
        angle_rads[:, 0::2]
    )

    encoding[:, 1::2] = np.cos(
        angle_rads[:, 1::2]
    )

    return tf.cast(
        encoding,
        dtype=tf.float32
    )


# Example
sequence_length = 5
d_model = 8

encoding = positional_encoding(
    sequence_length,
    d_model
)

print(encoding)

9. Understand the Python Code

Step 1 — Sequence Length

sequence_length = 5

This means our sequence contains 5 positions.

Position 0
Position 1
Position 2
Position 3
Position 4

Step 2 — Embedding Size

d_model = 8

Each token representation has 8 numbers.

[x1, x2, x3, x4, x5, x6, x7, x8]

Therefore, our positional encoding also needs 8 numbers for every position.

Step 3 — Create Positions

positions = np.arange(
    sequence_length
)[:, np.newaxis]

This creates:

[[0],
 [1],
 [2],
 [3],
 [4]]

These are the positions of the tokens.

Step 4 — Create Dimensions

dimensions = np.arange(
    d_model
)[np.newaxis, :]

This creates the embedding dimensions:

[[0, 1, 2, 3, 4, 5, 6, 7]]

Step 5 — Calculate Frequencies

angle_rates = 1 / np.power(
    10000,
    (2 * (dimensions // 2))
    / np.float32(d_model)
)

This determines how quickly the sine and cosine values change across different dimensions.

You don't need to memorize this calculation. It is the mathematical part of the positional encoding formula.

Step 6 — Calculate Angles

angle_rads = positions * angle_rates

Each position is combined with the corresponding frequency.

Step 7 — Create Encoding Matrix

encoding = np.zeros(
    (sequence_length, d_model)
)

With:

sequence_length = 5
d_model = 8

We create a matrix of:

5 × 8

So every one of the 5 positions gets an 8-number positional vector.

Step 8 — Apply Sine

encoding[:, 0::2] = np.sin(
    angle_rads[:, 0::2]
)

The even-numbered dimensions use sine.

0
2
4
6
...

Step 9 — Apply Cosine

encoding[:, 1::2] = np.cos(
    angle_rads[:, 1::2]
)

The odd-numbered dimensions use cosine.

1
3
5
7
...

Therefore:

Even dimensions → sin

Odd dimensions → cos

10. What Does the Encoding Look Like?

A simplified positional encoding matrix could look like:

Position 0
[0.00, 1.00, 0.00, 1.00, 0.00, 1.00]

Position 1
[0.84, 0.54, 0.05, 1.00, 0.00, 1.00]

Position 2
[0.91, -0.42, 0.10, 0.99, 0.00, 1.00]

Position 3
[0.14, -0.99, 0.15, 0.99, 0.00, 1.00]

Notice that each position has a different pattern.

Position 0 → unique pattern
Position 1 → unique pattern
Position 2 → unique pattern
Position 3 → unique pattern

These patterns tell the Transformer where the tokens are located.

11. Add Positional Encoding to Embeddings

Positional encoding is usually combined with token embeddings.

Token Embedding
       +
Positional Encoding
       ↓
Transformer Input

Example:

Token embedding:

[0.20, 0.50, 0.10, 0.80]


Position encoding:

[0.00, 1.00, 0.00, 1.00]

Add them:

[0.20, 0.50, 0.10, 0.80]
+
[0.00, 1.00, 0.00, 1.00]
--------------------------------
[0.20, 1.50, 0.10, 1.80]

The resulting vector contains both token information and positional information.

12. Same Word, Different Position

This is an important example.

Suppose the word:

"Python"

appears at different positions.

Python at position 0

Embedding
+
Position 0 encoding

Versus:

Python at position 5

Embedding
+
Position 5 encoding

The word embedding can be the same, but the final representation will be different because the positional encoding is different.

Same word
   +
Different position
   ↓
Different representation

13. Understanding Word Order

Consider:

I eat pizza.

Positions:

I       → 0
eat     → 1
pizza   → 2

Now change the order:

Pizza eat I.

Positions become:

Pizza   → 0
eat     → 1
I       → 2

Positional encoding lets the Transformer distinguish these different arrangements.

14. Positional Encoding Inside a Transformer

Text
 ↓
Tokenization
 ↓
Tokens
 ↓
Token Embeddings
 ↓
+
Positional Encoding
 ↓
Transformer Input
 ↓
Self-Attention
 ↓
Feed Forward Network
 ↓
Next Transformer Layer
 ↓
Output

Positional encoding is therefore added near the beginning of the Transformer.

15. What Positional Encoding Actually Tells the Model

It does not simply tell the model:

"This is word number 3."

Instead, it provides a numerical representation of position that the neural network can use when learning relationships between tokens.

Token
+
Position pattern
        ↓
Useful representation
        ↓
Attention can use it

16. Do All Modern Transformers Use This Exact Method?

No.

This is an important distinction.

The original Transformer architecture used sinusoidal positional encoding, but modern Transformer models can use other approaches to represent position.

For example, some architectures use learned positional embeddings or relative/rotary positional methods.

So don't make the mistake of thinking:

"Transformers always use sine and cosine."

The general requirement is:

Transformer
      ↓
Needs positional information

The exact technique can vary by architecture.

17. Complete Example

Sentence:

"I love Python"


        ↓


Tokens:

["I", "love", "Python"]


        ↓


Token Embeddings:

I       → [....]
love    → [....]
Python  → [....]


        ↓


Positions:

I       → 0
love    → 1
Python  → 2


        ↓


Positional Encoding:

Position 0 → [....]
Position 1 → [....]
Position 2 → [....]


        ↓


Add:

Embedding + Position


        ↓


Transformer Input


        ↓


Self-Attention

18. Final Summary

Problem:

Transformer processes tokens in parallel.

Therefore:

It needs explicit position information.


Solution:

Positional Encoding


Formula:

Embedding + Position Encoding


Result:

The model knows:

WHAT the token is
+
WHERE the token is

The easiest way to remember it:

Token Embedding
      +
Positional Encoding
      =
Transformer Input

And the core idea is:

Embedding tells the model:
"What is this?"

Positional encoding tells the model:
"Where is it?"