Security

What Is GenAI Security?

GenAI security is the set of application, infrastructure, data, and model controls used to keep an AI system confidential, authorized, reliable, and safe.

What Are You Protecting?

Start by identifying the assets an attacker could access or manipulate.

• Customer data: Personal, account, order, or support information.
• Business data: Internal documents, pricing, prompts, and proprietary knowledge.
• Credentials: API keys, database passwords, tokens, and service identities.
• Tools: Email, payments, databases, search, and business APIs.
• Model context: System instructions, retrieved content, and conversation state.
• Availability: Compute, API quotas, and services that can be exhausted.

Threat Model the AI Application

Security design becomes clearer when you map trust boundaries and ask what can go wrong at each boundary.

Authentication and Authorization

Users should authenticate before accessing protected AI features, but authentication alone is not enough. Authorization must determine what each user is allowed to see and do.

Protect Secrets

Never place API keys, database passwords, private tokens, or other secrets directly in prompts, source code, client-side JavaScript, or model-visible context.

import os

# Read secrets from the server environment.
api_key = os.environ["OPENAI_API_KEY"]

# Use api_key in a server-side SDK call.
# Never send the key to the browser or the model.
print("API key loaded:", bool(api_key))

Use a secrets manager or protected environment configuration in production, with rotation and access auditing.

Prompt Injection

Prompt injection happens when untrusted content attempts to change the model's intended behavior. This can happen through direct user prompts or indirectly through documents, websites, emails, and retrieved knowledge.

Defend Against Prompt Injection

Do not rely on one magic prompt sentence. Use multiple application-level controls.

Secure Tool Calling

Tool calling creates a direct bridge from model output to real systems. The model can propose an action, but your application must authorize and validate that action.

Python: Validate a Tool Argument

Validate sensitive arguments in normal application code before a tool can execute.

ALLOWED_STATUSES = {"open", "closed"}


def update_ticket(ticket_id: int, status: str, user_can_edit: bool):
    if not user_can_edit:
        raise PermissionError("User is not allowed to edit tickets")
    if status not in ALLOWED_STATUSES:
        raise ValueError("Invalid ticket status")

    return {"ticket_id": ticket_id, "status": status}

Schema validation is useful, but authorization and business rules must still run before execution.

Protect Sensitive Data

Minimize the amount of sensitive data sent to a model. If the model does not need a field, do not include it.

RAG Security

Retrieval improves knowledge access but introduces another trust boundary. A retrieved document should not automatically gain permission to instruct the model or reveal data.

Multi-Tenant Data Isolation

For SaaS and B2B applications, a user from one tenant must never retrieve another tenant's private documents.

Output Validation

Model output is untrusted too. Before displaying, storing, or executing it, validate its structure and apply application-specific rules.

Python: Safe Output Handling

Keep model-generated text as data until your application explicitly decides what to do with it.

def accept_summary(summary: str) -> str:
    if not isinstance(summary, str):
        raise TypeError("Expected text")

    cleaned = summary.strip()
    if len(cleaned) > 2000:
        raise ValueError("Summary is too long")

    # Treat the result as data; do not execute it as code.
    return cleaned

Rate Limits and Abuse Prevention

Attackers can abuse expensive model calls even without stealing data. Protect endpoints with authentication, rate limits, quotas, request-size limits, and sensible timeouts.

Logging and Observability

Security events need enough evidence for investigation, but logs must not become another data-leak channel.

Dependency and Infrastructure Security

Secure the surrounding system as well as the model layer. Keep Python packages, containers, operating systems, SDKs, databases, and cloud permissions up to date and controlled.

• Patch dependencies: Track vulnerabilities and update supported versions.
• Use isolated services: Separate high-risk tools from the main application where appropriate.
• Restrict network access: Allow only required outbound and inbound connections.
• Rotate credentials: Limit the lifetime and scope of secrets.

Human Approval for High-Risk Actions

Some operations should not be fully autonomous. For example, deleting records, sending sensitive communications, issuing refunds, or changing critical configuration may require explicit approval.

Security Testing

Test the complete application, not just the prompt. Include normal users, malicious inputs, unexpected documents, invalid tool arguments, permission boundaries, and data-isolation checks.

Security Architecture Checklist

Common Security Mistakes

• Trusting the prompt: Treating model instructions as an authorization mechanism.
• Giving broad tool access: Allowing a model to call powerful APIs without checks.
• Ignoring retrieved instructions: Assuming RAG documents are always safe.
• Logging everything: Accidentally storing passwords, tokens, or private customer data.
• No tenant filter: Relying on the model instead of enforcing isolation in storage/retrieval.
• Executing output: Passing model-generated strings directly into sensitive operations.

Mini Project: Secure Support Agent

Design a customer-support agent that can search approved knowledge, inspect a customer's own order, and draft a response.

Quiz