Pre-training

Pre-training in one picture

Think of pre-training as building the model's general language foundation. The model is not initially taught “answer customer emails” or “write Python.” It is exposed to enormous amounts of token sequences and learns statistical relationships that can later support many tasks.

Why do we pre-train a model?

Starting from random parameters gives the model no useful language knowledge. Pre-training lets it discover patterns such as word relationships, syntax, common facts, code structures, and relationships between pieces of text.

Important: pre-training does not guarantee truth. A model learns patterns from its training data; it does not receive a built-in database of verified facts.

Step 1: build a training corpus

A pre-training run starts with a very large collection of data. A production pipeline may combine multiple sources, apply quality filters, remove duplicates, manage permissions, and construct a mixture with deliberate proportions.

Step 2: turn text into prediction examples

The tokenizer maps text to token IDs. A sequence can then be shifted by one position so the model receives an input and a target at every position.

One sequence therefore supplies several training positions. Across millions or billions of sequences, this creates an enormous number of prediction opportunities.

The pre-training objective: next-token prediction

At each position, the model produces probabilities for possible next tokens. The training target tells it which token actually followed in the dataset. The loss function turns the difference into a number that optimization can minimize.

If the target token receives a high probability, the loss is relatively small. If the model assigns it very little probability, the loss is larger and the gradient provides a stronger learning signal.

Step 3: repeat the learning loop

Pre-training is not one pass through the corpus. The system processes batches, runs forward passes, calculates loss, backpropagates gradients, updates parameters, records metrics, and periodically saves checkpoints.

Why scale changes everything

The learning algorithm can remain conceptually simple while the engineering becomes enormous. Larger models, more tokens, and more compute can improve capability, but they also increase memory, communication, storage, and reliability requirements.

Build a tiny pre-training model in Python

We can reproduce the core idea on a tiny scale. This model learns to predict the next word from the previous word. It is not an LLM, but the objective mirrors the essential next-token learning loop.

import torch
from torch import nn

sentences = [
    "cats chase mice",
    "cats like milk",
    "dogs chase balls",
    "dogs like walks",
]

words = sorted({word for sentence in sentences for word in sentence.split()})
word_to_id = {word: i for i, word in enumerate(words)}

pairs = []
for sentence in sentences:
    ids = [word_to_id[word] for word in sentence.split()]
    pairs.extend((ids[i], ids[i + 1]) for i in range(len(ids) - 1))

x = torch.tensor([a for a, _ in pairs])
y = torch.tensor([b for _, b in pairs])

model = nn.Sequential(
    nn.Embedding(len(words), 16),
    nn.Linear(16, len(words))
)

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.03)

for step in range(400):
    logits = model(x)
    loss = loss_fn(logits, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

context = torch.tensor([word_to_id["cats"]])
probs = model(context).softmax(dim=-1)[0]

for word, score in sorted(zip(words, probs.tolist()), key=lambda item: item[1], reverse=True):
    print(f"{word:8} {score:.3f}")

Change the data and watch the learned distribution change

This is one of the most useful pre-training experiments: change the corpus, train again, and compare the model's predictions. The learned behavior changes because the training signal changed.

Pre-training vs fine-tuning vs instruction tuning

These stages can be combined in different system designs, but they answer different questions: What broad patterns can the model learn? → How should it adapt? → How should it respond to instructions?

Common misconceptions

Mini-project: build a tiny pre-training experiment

Use the notebook with this lesson to train the small next-word model and then deliberately change its training corpus.