RLHF

Why do we need RLHF?

Instruction tuning teaches a model to follow examples, but many possible answers can satisfy the same instruction. People may prefer one answer because it is clearer, more complete, more helpful, more concise, or safer.

Start with a real preference decision

Imagine a customer asks: “My order is five days late. What should I do?” Both answers are understandable, but a support team may prefer one.

Notice that the human is not writing a new answer. They are providing a preference signal between candidate answers.

What is human preference data?

A common setup gives a reviewer the same prompt and two or more candidate responses. The reviewer ranks them or chooses the preferred response.

example = {
    "prompt": "Explain a refund policy simply.",
    "response_a": "You can request a refund within 30 days.",
    "response_b": "Refunds may be available under the policy.",
    "preferred": "response_a",
}

print("Human preferred:", example["preferred"])

Step 1: start from an instruction-following model

In the classic RLHF pipeline, the model usually begins with broad pre-training and supervised fine-tuning (SFT). SFT provides examples of instructions and high-quality responses so the model has a useful assistant-like starting point.

RLHF is therefore not a replacement for pre-training. It is a later stage that uses preference information to further shape behavior.

Step 2: train a reward model

A reward model learns to predict which candidate response would receive a higher human preference score. Instead of asking a person to judge every response during optimization, the learned reward model provides a scalable approximation of those judgments.

Step 3: optimize the language model with the reward signal

In the classic RLHF formulation, the language model generates responses, the reward model scores them, and an RL algorithm updates the policy while regularizing it so it does not move too far from the starting model.

The loop is repeated over many examples. The goal is not simply to maximize any arbitrary score; practical systems also use constraints and evaluation to reduce reward hacking and preserve useful model capabilities.

Build a tiny preference scorer in Python

Training a production reward model requires a neural network and a large preference dataset. We can still understand the core idea with a tiny transparent experiment: define a few measurable signals and combine them into a simple score.

responses = [
    {"name": "A", "helpful": 0.9, "clear": 0.8},
    {"name": "B", "helpful": 0.6, "clear": 0.9},
]

def score(response):
    return 0.7 * response["helpful"] + 0.3 * response["clear"]

for response in responses:
    response["reward"] = score(response)

best = max(responses, key=lambda item: item["reward"])
print(best)

What this demonstrates: a scoring function can turn several desired properties into one ranking signal. A real reward model learns such a relationship from preference data rather than using hand-written weights like this toy example.

Why RLHF is difficult

Modern alignment systems can use methods beyond the classic RLHF pipeline, including direct preference optimization and other preference-learning approaches. The common idea is still to use information about preferred versus less-preferred outputs.

RLHF vs other techniques