Prompt Optimization

What Is Prompt Optimization?

Prompt optimization is the process of improving a prompt so that an AI application performs better on the task it is designed to solve.

“Better” needs to be defined. Depending on the application, you may care about correctness, consistency, format compliance, helpfulness, safety, cost, or response time.

Try a longer instruction.
Try a different adjective.
Try another example.
Looks better → ship it.

Baseline prompt
      ↓
Test dataset
      ↓
Measure results
      ↓
Change one thing
      ↓
Compare again

Start With a Clear Optimization Goal

Before changing a prompt, decide what you are trying to improve. A prompt cannot be “better” in the abstract.

Create a Baseline Prompt

Your first tested version is the baseline. Every later version should be compared against it using the same evaluation conditions.

Classify the customer message.

Categories:
- billing
- shipping
- technical
- account

Return the category.

Message:
{message}

Save the prompt text, model settings, test inputs, outputs, and evaluation results. Without a baseline, you cannot confidently say that a change helped.

Build an Evaluation Dataset

A small evaluation dataset is a collection of inputs with expected outcomes. It lets you compare prompt versions fairly.

Measure More Than “Looks Good”

Choose metrics that match your task. For classification, accuracy and invalid-output rate may be useful. For extraction, field-level correctness matters. For generation, a human or model-based rubric may be needed.

Change One Important Thing at a Time

If you replace the instruction, add five examples, change the output format, and change the model at the same time, you will not know which change caused the result.

Controlled changes make prompt optimization easier to debug and explain to other developers.

Improve the Prompt in Layers

A useful optimization sequence is to remove ambiguity before adding complexity.

Before and After: Optimize a Classifier

Classify this message.

Message:
"I need to change my email."

TASK:
Classify the message.

CATEGORIES:
billing, shipping, technical, account

RULES:
- Choose exactly one category.
- Use only the supplied message.
- Return the category in lowercase.
- Do not add extra text.

MESSAGE:
"I need to change my email."

OUTPUT:

Python Example: Compare Prompt Versions

A simple evaluation loop can compare two prompt versions on the same test inputs.

from openai import OpenAI

client = OpenAI()

test_cases = [
    ("I was charged twice.", "billing"),
    ("My tracking has not changed.", "shipping"),
    ("The app crashes.", "technical"),
    ("I cannot reset my password.", "account"),
]

prompt_v1 = """
Classify the message as billing, shipping, technical, or account.
Return the category.

Message:
{message}
"""

prompt_v2 = """
TASK:
Classify the message.

CATEGORIES:
billing, shipping, technical, account

RULES:
- Choose exactly one category.
- Return only the lowercase category.
- Do not add an explanation.

MESSAGE:
{message}

OUTPUT:
"""

def classify(template, message):
    response = client.responses.create(
        model="YOUR_MODEL",
        input=template.format(message=message),
    )
    return response.output_text.strip().lower()

for name, template in [("v1", prompt_v1), ("v2", prompt_v2)]:
    correct = 0

    for message, expected in test_cases:
        prediction = classify(template, message)

        if prediction == expected:
            correct += 1

    accuracy = correct / len(test_cases)
    print(name, accuracy)

Optimize Few-shot Examples Too

Prompt optimization also applies to demonstrations. More examples can improve coverage, but they consume context and may add latency and cost.

Watch for Prompt Regressions

An optimization can fix one case while breaking another. This is a prompt regression.

The overall score improved, but an important subgroup became worse. Keep separate checks for critical categories and edge cases rather than relying only on one aggregate number.

Optimize for Quality, Cost, and Latency

Production prompt optimization is usually a trade-off. A longer prompt may improve quality but increase token usage. A shorter prompt may be cheaper but less reliable.

Prompt Optimization in Production

Once a prompt is used by real users, optimization becomes an ongoing engineering process.