Self-Attention

1. What Is Self-Attention?

Remember the basic idea of attention:

Attention
    ↓
Find important information
    ↓
Use that information
    ↓
Produce a better representation

Self-attention applies this idea within the same sequence.

Input sequence
      ↓
Every token looks at
other tokens in the same sequence
      ↓
Determine relationships
      ↓
Create contextual representations

That is why it is called self-attention.

2. Simple Example

Consider this sentence:

The cat sat on the mat.

Imagine we are processing the word "cat".

The
 ↓
cat
 ↓
sat
 ↓
on
 ↓
the
 ↓
mat

Self-attention allows the representation of "cat" to consider information from the other words.

For example, "sat" may be useful because it tells us something about what the cat is doing.

cat
 ↓
looks at

The
sat
on
the
mat

The model learns which relationships are useful rather than treating every word as equally important.

3. Every Token Can Attend to Other Tokens

This is the most important difference to understand.

Token 1 ───────→ Token 2
   │                 │
   ↓                 ↓
Token 3 ←──────── Token 4
   │
   ↓
Token 5

Conceptually, each token can calculate how much attention it should give to other tokens.

For example:

The → looks at other tokens
cat → looks at other tokens
sat → looks at other tokens
on  → looks at other tokens
the → looks at other tokens
mat → looks at other tokens

This produces contextual representations for all tokens.

4. Self-Attention Uses Context

Consider:

I went to the bank to deposit money.

The word "bank" is strongly related to words such as:

bank
 ↓
deposit
money

Now consider:

We sat on the bank of the river.

Here the context is different:

bank
 ↓
river
sat

The same word can therefore receive a different contextual representation depending on the surrounding sequence.

5. Query, Key, and Value

Self-attention uses three representations:

Query
Key
Value

A simple mental model is:

Query
"What am I looking for?"

Key
"What information do I contain?"

Value
"What information should I provide?"

For every input token, the model creates a query, key, and value representation.

Token
  ↓
 ┌───────────────┐
 │       │       │
 Query   Key   Value
 └───────────────┘

6. How Self-Attention Works

Suppose we have:

The cat sat.

For each token, the model creates Q, K, and V.

The → Q₁ K₁ V₁
cat → Q₂ K₂ V₂
sat → Q₃ K₃ V₃

Now suppose we are processing "cat".

Q₂
 ↓
Compare with

K₁
K₂
K₃

This gives attention scores.

Q₂ · K₁ → score
Q₂ · K₂ → score
Q₂ · K₃ → score

The scores are converted into attention weights.

scores
   ↓
softmax
   ↓
weights

Finally, the values are combined using those weights.

weights
   ↓
weighted V₁
weighted V₂
weighted V₃
   ↓
combined result
   ↓
new representation for "cat"

7. Attention Between All Tokens

For a sentence containing several tokens, we can think of attention as a matrix.

          The   cat   sat   mat

The       0.2   0.4   0.1   0.3
cat       0.1   0.5   0.3   0.1
sat       0.1   0.4   0.4   0.1
mat       0.2   0.1   0.2   0.5

These numbers are only an educational example.

Each row represents one token asking:

"How much attention should I give
to each token?"

For example:

cat
 ↓
The   → 0.1
cat   → 0.5
sat   → 0.3
mat   → 0.1

Here, the model is giving the highest attention weight to "cat" itself and some attention to "sat".

8. Self-Attention Formula

The standard self-attention calculation is:

SelfAttention(Q, K, V)
=
softmax(QKᵀ / √dₖ)V

Understand it as four steps:

Step 1
QKᵀ
 ↓
Calculate similarity


Step 2
÷ √dₖ
 ↓
Scale the scores


Step 3
softmax(...)
 ↓
Create attention weights


Step 4
weights × V
 ↓
Create output

9. Simple Numerical Example

Let's make the calculation easier by using small vectors.

Query:

Q = [1, 0]


Keys:

K₁ = [1, 0]
K₂ = [0, 1]
K₃ = [1, 1]

Calculate the dot products:

Q · K₁

[1, 0] · [1, 0]

= 1

Q · K₂

[1, 0] · [0, 1]

= 0

Q · K₃

[1, 0] · [1, 1]

= 1

Therefore:

Scores = [1, 0, 1]

Apply softmax:

Softmax([1, 0, 1])

≈ [0.42, 0.16, 0.42]

Now the model has attention weights.

K₁ → 0.42
K₂ → 0.16
K₃ → 0.42

The second key receives less attention because its similarity score was lower.

10. Build Self-Attention With Python

Now let's implement a small self-attention calculation using TensorFlow.

import tensorflow as tf


# Three tokens
inputs = tf.constant([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0]
])


# In a real Transformer these are
# produced using trainable weight matrices.

Q = inputs
K = inputs
V = inputs


# Calculate attention scores
scores = tf.matmul(
    Q,
    K,
    transpose_b=True
)


# Scale scores
d_k = tf.cast(tf.shape(K)[-1], tf.float32)

scaled_scores = scores / tf.sqrt(d_k)


# Convert scores to attention weights
weights = tf.nn.softmax(
    scaled_scores,
    axis=-1
)


# Weighted sum of values
output = tf.matmul(
    weights,
    V
)


print("Scores:")
print(scores.numpy())

print("\nAttention weights:")
print(weights.numpy())

print("\nOutput:")
print(output.numpy())

11. Understand the Python Code

Step 1 — Input Tokens

inputs = tf.constant([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0]
])

We have three tokens.

Token 1 → [1, 0]
Token 2 → [0, 1]
Token 3 → [1, 1]

These are simplified token representations. Real models use much larger embedding vectors.

Step 2 — Q, K, V

Q = inputs
K = inputs
V = inputs

We are deliberately using the same values here to make the concept easy to understand.

In a real Transformer, Q, K, and V are normally created from the input using learned transformations.

Step 3 — Calculate Scores

scores = tf.matmul(
    Q,
    K,
    transpose_b=True
)

This compares every query with every key.

Q₁ compares with K₁, K₂, K₃
Q₂ compares with K₁, K₂, K₃
Q₃ compares with K₁, K₂, K₃

Therefore we get a matrix.

          K₁    K₂    K₃

Q₁        ?     ?     ?
Q₂        ?     ?     ?
Q₃        ?     ?     ?

Step 4 — Scale

d_k = tf.cast(
    tf.shape(K)[-1],
    tf.float32
)

scaled_scores = scores / tf.sqrt(d_k)

We divide by the square root of the key dimension. This keeps the attention scores better behaved.

Step 5 — Softmax

weights = tf.nn.softmax(
    scaled_scores,
    axis=-1
)

Softmax converts each row of scores into attention weights.

Scores
   ↓
Softmax
   ↓
Weights
   ↓
Each row approximately adds up to 1

Step 6 — Weighted Values

output = tf.matmul(
    weights,
    V
)

The model now combines the value vectors according to the attention weights.

Attention weights
        +
Value vectors
        ↓
Weighted combination
        ↓
Output representation

12. Real Transformer Self-Attention

The previous Python example intentionally simplified the process.

A real Transformer does not simply use:

Q = inputs
K = inputs
V = inputs

Instead, it learns transformations for Q, K, and V.

Input
  │
  ├──→ Linear transformation → Q
  │
  ├──→ Linear transformation → K
  │
  └──→ Linear transformation → V

These transformations contain trainable parameters.

Conceptually:

Q = XWQ
K = XWK
V = XWV

Where:

X
 ↓
Input representations

WQ
 ↓
Query weights

WK
 ↓
Key weights

WV
 ↓
Value weights

13. Self-Attention With Keras

In practical TensorFlow projects, we can use MultiHeadAttention.

import tensorflow as tf
from tensorflow.keras import layers


# Batch = 2
# Tokens = 5
# Features = 16

inputs = tf.random.normal(
    (2, 5, 16)
)


attention = layers.MultiHeadAttention(
    num_heads=2,
    key_dim=16
)


outputs = attention(
    query=inputs,
    key=inputs,
    value=inputs
)


print("Input:")
print(inputs.shape)


print("Output:")
print(outputs.shape)

Because:

query = inputs
key   = inputs
value = inputs

the layer is performing self-attention.

14. Why Is Self-Attention Powerful?

The biggest advantage is that a token can directly interact with other tokens in the sequence.

Token 1 ────────────────→ Token 5
   ↑                          ↓
   │                          │
   └──────── Token 3 ─────────┘

Information does not have to be passed through a long chain of recurrent hidden states like in a basic RNN.

For example:

The student who studied hard
passed the difficult examination.

"student"
     ↕
"studied"
     ↕
"passed"
     ↕
"examination"

Self-attention can learn relationships between these tokens directly.

15. Self-Attention vs RNN

RNN

Token 1
   ↓
Token 2
   ↓
Token 3
   ↓
Token 4
   ↓
Token 5


Self-Attention

Token 1 ──┐
Token 2 ──┤
Token 3 ──┼──→ Attention
Token 4 ──┤
Token 5 ──┘

An RNN processes the sequence recurrently.

Self-attention calculates relationships between tokens directly.

This is one of the fundamental ideas that made Transformers so successful.

16. Important Limitation

Self-attention is powerful, but it is not free.

Every token can compare itself with many other tokens.

10 tokens
   ↓
100 pairwise comparisons


1,000 tokens
   ↓
1,000,000 pairwise comparisons

The attention calculation has roughly quadratic scaling with sequence length:

Sequence length = n

Attention score matrix
≈ n × n

Complexity
≈ O(n²)

This becomes expensive for extremely long sequences. That is why efficient-attention techniques are an important research area.

17. Complete Self-Attention Flow

Input sequence
      ↓
Token representations
      ↓
Create Q, K, V
      ↓
Q × Kᵀ
      ↓
Attention scores
      ↓
Scale by √dₖ
      ↓
Softmax
      ↓
Attention weights
      ↓
Multiply by V
      ↓
Context-aware representations
      ↓
Next Transformer operation

18. Simple Way to Remember It

Self-Attention means:

"I am one token.

I will look at the other tokens
in my own sequence.

I will decide which ones are
important to me.

Then I will combine their
information to create a better
representation of myself."

That is the core intuition.

19. Final Summary

Self-Attention

1. Every token creates Q, K, V.

2. Each Query compares with all Keys.

3. Similarity scores are calculated.

4. Scores are scaled.

5. Softmax converts scores into weights.

6. Weights are applied to Values.

7. The weighted Values produce
   contextual representations.

8. Every token gets its own
   context-aware representation.

The one sentence you should remember is:

Self-attention allows every token
to look at every other token in
the same sequence and decide
how much information to use.