Monitoring

What Is Monitoring?

Monitoring is the continuous collection and analysis of signals from your AI application so the team can understand system health and user experience.

Why GenAI Monitoring Is Different

Traditional web applications often have deterministic outputs. GenAI systems introduce probabilistic behavior, model changes, retrieval quality, tool decisions, and variable token usage.

• Variable latency: Response time changes with model, context, and output length.
• Variable cost: Token usage can change from request to request.
• Quality drift: A technically successful response can become less useful.
• Retrieval failures: The model may receive incomplete or irrelevant context.
• Tool failures: External APIs can fail independently of the model.
• Model changes: Provider updates can change behavior without application code changes.

What Should You Monitor?

A practical monitoring strategy combines infrastructure, application, model, retrieval, security, and quality signals.

Logs, Metrics, and Traces

These three telemetry types answer different questions and work best together.

Design a Useful Request ID

Give each user request a correlation ID so a support engineer can follow the request across services.

from uuid import uuid4


def start_request(user_id: str) -> dict:
    return {
        "request_id": str(uuid4()),
        "user_id": user_id,
    }


context = start_request("user-123")
print(context["request_id"])

Use the request ID in application logs, model-call records, tool traces, and user-facing support workflows.

Monitor Latency

Track latency as a distribution rather than only an average. p50 shows the typical request, while p95 and p99 expose slow experiences.

Monitor Errors and Retries

Separate error types so an increasing error rate leads to the correct action.

Also track retry counts. A request that eventually succeeds after several retries may hide a reliability problem and increase cost and latency.

Monitor Token Usage and Cost

Token usage is one of the most important operational signals for GenAI applications.

• Input tokens per request
• Output tokens per request
• Total tokens by model
• Average cost per request
• Cost by feature or tenant
• Daily and monthly spend

Python: Record Request Metrics

A simple application can start by recording duration, token counts, and status for every request.

import time


def observe_request(run_model):
    started = time.perf_counter()
    status = "success"

    try:
        result = run_model()
        return result
    except Exception:
        status = "error"
        raise
    finally:
        duration_ms = (time.perf_counter() - started) * 1000
        print({
            "status": status,
            "duration_ms": round(duration_ms, 2),
        })

In production, send these measurements to your telemetry system instead of printing them to stdout.

Monitor Retrieval Quality

For RAG systems, monitoring the model alone is not enough. You also need to understand what context was retrieved.

Keep enough trace metadata to investigate retrieval failures, but avoid storing sensitive document contents unnecessarily.

Monitor Tool Calls

Agents can create operational risk because one user request may trigger several external actions.

Monitor AI Quality

Technical health does not prove that the answer is useful. Add quality signals appropriate to the application.

Online vs Offline Evaluation

Use both real production signals and controlled evaluation datasets.

Monitoring Guardrails and Safety

Record guardrail outcomes so you can identify attacks, false positives, and policy changes.

• Blocked input count
• Blocked output count
• Prompt injection detections
• PII redaction events
• Tool approval requests
• Human escalations

What Belongs in a Trace?

A useful trace captures enough metadata to reconstruct a request without turning observability into a copy of your entire data store.

Alerting: When Should You Be Notified?

Alerts should point to actionable problems rather than every small fluctuation.

Useful alerts include high error rates, provider failures, abnormal spend, latency regressions, retrieval failures, and spikes in safety events.

Dashboards That Matter

A production dashboard should make the health of the application understandable in seconds.

Monitoring a Multi-Agent System

For multi-agent applications, one request can become a tree of agent calls. Track parent and child spans so you can see which agent caused latency, cost, or failure.

Detecting Drift

Monitor changes in traffic, prompts, retrieved sources, model behavior, and user outcomes over time.

Common Monitoring Mistakes

• Monitoring only server uptime
• Tracking average latency but not p95/p99
• Ignoring token and cost anomalies
• Logging sensitive prompts and documents by default
• Collecting logs without request correlation IDs
• Measuring model calls but not retrieval or tools
• Alerting on everything without clear actions
• Never connecting technical metrics to user outcomes

Mini Project: Build a GenAI Monitoring Layer

Create a small Python wrapper around your AI pipeline that records a request ID, latency, status, token usage, retrieval count, tool-call count, and quality feedback.

from dataclasses import dataclass
from time import perf_counter
from uuid import uuid4


@dataclass
class GenAIMetrics:
    request_id: str
    duration_ms: float
    status: str
    retrieved_docs: int
    tool_calls: int


def monitor(run_pipeline, retrieved_docs=0, tool_calls=0):
    request_id = str(uuid4())
    started = perf_counter()
    status = "success"

    try:
        result = run_pipeline()
        return result
    except Exception:
        status = "error"
        raise
    finally:
        metrics = GenAIMetrics(
            request_id=request_id,
            duration_ms=(perf_counter() - started) * 1000,
            status=status,
            retrieved_docs=retrieved_docs,
            tool_calls=tool_calls,
        )
        print(metrics)

Extend this pattern with your telemetry backend, token counts, model metadata, trace spans, and privacy-safe quality signals.

Monitoring Checklist

Production Monitoring Architecture

A mature system connects application telemetry to dashboards, alerts, evaluation, and an improvement loop.

Monitoring should feed engineering decisions: tune prompts, change retrieval, optimize models, fix tools, improve guardrails, and update evaluation sets.

Quick Quiz