Latency

What Is Latency?

Latency is the time between an action being requested and the useful result becoming available. In a GenAI application, it is usually measured from the user's request until the answer starts or finishes arriving.

TTFT vs Total Response Time

Two measurements are especially useful for streaming AI applications.

Where Does Latency Come From?

A production request can contain several latency components.

• Network: Browser, API gateway, model provider, and database round trips.
• Queueing: Waiting for an available worker or overloaded service.
• Retrieval: Vector search, metadata filtering, and reranking.
• Model: Time to start and generate output tokens.
• Tools: APIs, databases, search, or business systems.
• Retries: Failed calls can multiply the total request time.

Measure Before You Optimize

Start with a latency budget and measure every important stage. Do not optimize based only on how fast the model provider says the model is.

Python: Measure a Stage

Simple timing instrumentation helps identify which operation is actually slow.

from time import perf_counter

start = perf_counter()

# Replace this with a real retrieval or model call.
result = "retrieved context"

elapsed_ms = (perf_counter() - start) * 1000
print(f"Stage latency: {elapsed_ms:.1f} ms")

In production, attach the measurement to a request ID so you can compare stages across real traffic.

Reduce Sequential Work

One of the biggest latency problems is performing independent operations one after another.

Parallelize Independent Calls

If two operations do not depend on each other, run them concurrently. This can reduce wall-clock time even when the individual operations are unchanged.

Python: Parallel Async Work

Python's async tools can start independent I/O operations together.

import asyncio

async def retrieve_docs():
    await asyncio.sleep(0.3)
    return "docs"

async def load_customer():
    await asyncio.sleep(0.3)
    return "customer"

async def prepare_context():
    docs, customer = await asyncio.gather(
        retrieve_docs(),
        load_customer(),
    )
    return docs, customer

print(asyncio.run(prepare_context()))

The example simulates I/O. Real applications should use async-compatible clients for the services they call.

Streaming Makes AI Feel Faster

Streaming sends generated output to the user as it becomes available instead of waiting for the complete answer.

Choose Faster Models When Quality Allows

Model capability, output length, and generation speed often trade off against one another. Select a model based on the quality target and measured latency, not marketing labels alone.

Control Output Length

Generating more tokens takes more time. Ask for the amount of output the UI actually needs.

Optimize RAG Retrieval

Retrieval should return enough evidence for a correct answer without forcing the system to process unnecessary data.

• Use appropriate chunk sizes.
• Use metadata filters when the query has known constraints.
• Keep the candidate set reasonable.
• Use reranking only when its quality benefit justifies its latency.
• Send focused context to the model instead of whole documents.
• Measure retrieval latency separately from generation latency.

Cache Repeated Work

Caching can remove entire model or retrieval calls from the critical path when the same result can safely be reused.

Keep Context Small

Large prompts can increase processing time and make every request heavier. Keep only information that is relevant to the current task.

Agent Latency Can Multiply

Agents may call a model and tools several times. A five-step sequential agent can feel much slower than a single model request.

Queueing and Background Work

Not every AI task must block the user request. Reports, document ingestion, bulk embeddings, evaluations, and notifications can run asynchronously.

Timeouts and Retries

Retries can improve reliability but can also create large latency spikes. Use bounded retries, timeouts, and backoff.

Latency Budgets

Set a target for the complete user journey and divide it among the important stages.

These numbers are an example budget, not a universal benchmark. Choose targets from your product requirements and real measurements.

Perceived vs Actual Latency

Users care about how quickly the interface becomes useful. A progress state, streaming response, or immediate acknowledgment can improve perceived responsiveness while the full operation continues.

Latency and Cost Are Connected

Performance improvements can sometimes reduce cost because fewer retries, smaller prompts, fewer model calls, and less infrastructure work are required. But faster is not automatically cheaper.

Production Observability

Track latency by endpoint, feature, model, customer segment, and workflow. Useful measurements include TTFT, total response time, stage duration, timeout rate, retry count, and error rate.

Common Latency Mistakes

• Optimizing average latency while ignoring p95 and p99.
• Calling independent APIs sequentially.
• Sending huge prompts and entire documents.
• Using reranking or tools without measuring their benefit.
• Allowing unlimited agent loops.
• Retrying every error without a timeout or backoff.
• Measuring model time but ignoring retrieval and network time.
• Improving perceived speed while leaving the actual system unreliable.

Mini Project: Build a Faster RAG Assistant

Quick Quiz

30-Second Recap