LLM Security

Why LLM security is a distinct discipline

Large language models introduce security properties that do not appear in conventional application security:

• Prompt as attack surface: The natural language input to an LLM is also the primary attack vector. Malicious inputs can cause models to reveal training data, generate harmful content, bypass instructions, or exfiltrate information — without exploiting any code-level vulnerability.
• Opaque reasoning: Unlike a deterministic API, an LLM reasons over its input in ways that are not fully auditable from the output alone. The same input can produce different outputs; the same attack may succeed in one session and fail in another.
• Indirect input sources: Applications that retrieve external content — web pages, documents, emails — and provide it to an LLM create a path for adversarial content to influence the model’s behavior even when the model itself is not the direct target of the attack.
• Multi-tenant exposure: Organizations deploying LLMs on shared infrastructure may be exposed to prompt injection attacks, data cross-contamination, or inference-time leakage across user sessions.

The core threat categories in LLM security

Prompt injection

Prompt injection is the leading LLM security threat, ranked first in the OWASP LLM Top 10. An attacker embeds malicious instructions in content that the model processes — a document, email, or retrieved web page — causing the model to follow those instructions instead of its original task. Direct prompt injection comes from the user; indirect prompt injection is embedded in external content the model retrieves autonomously.

Training data extraction

LLMs can be prompted to reproduce memorized training data, including personally identifiable information, copyrighted text, or confidential content from the training corpus. Extraction attacks use carefully crafted prompts designed to surface memorized material and are particularly relevant for models fine-tuned on proprietary or regulated datasets.

Insecure output handling

LLMs integrated with downstream systems — code interpreters, web browsers, database interfaces — may generate outputs that are executed as code or queries without proper validation. LLM-generated content that reaches a browser unsanitized can trigger cross-site scripting (XSS); content passed to a database interface without validation can enable injection attacks.

Data leakage via context window

In Retrieval-Augmented Generation (RAG) systems and multi-turn conversations, sensitive documents may persist in the model’s context window and be inadvertently surfaced in later responses. Without session isolation and context boundary controls, one user’s sensitive data can appear in another user’s session.

Model inversion and membership inference

Model inversion attacks attempt to reconstruct sensitive properties of training data from model outputs. Membership inference attacks determine whether a specific data sample was included in training. Both are relevant for organizations that fine-tune models on internal or regulated datasets.

LLM security vs. AI agent security

LLM security and AI agent security are related but distinct disciplines:

Agentic AI risk builds on LLM security: an agent that is vulnerable to prompt injection and has real-world tool access is a significantly higher-severity threat than an LLM answering questions in isolation.

How LLM security controls work

LLM security controls operate at several layers of the application stack:

Input controls

• Prompt sanitization: Inspect user inputs and retrieved content for known prompt injection patterns before forwarding to the model.
• System prompt protection: Treat the system prompt as a security boundary; do not expose it in error messages or allow user inputs to override it.
• Input schema enforcement: Where possible, constrain inputs to expected formats to reduce the surface area for adversarial manipulation.

Output controls

• PII and sensitive data scanning: Scan model outputs before returning them to users or passing them to downstream systems.
• Output schema validation: Where models generate structured outputs (JSON, code), validate against a strict schema before execution.
• Content policy enforcement: Filter outputs for policy violations at the infrastructure layer, independent of model safety training.

Access controls

• Least-privilege data access: Limit the data a model can retrieve in RAG configurations to the minimum required for the current user’s authorization scope.
• Session isolation: Ensure that conversational context and retrieved documents do not persist across user sessions.
• Credential scoping: For models with tool access, issue session-scoped credentials rather than long-lived service account keys.

Audit and observability

• Maintain a tamper-evident log of every LLM interaction: the input, the model used, the output, and any policy decisions applied.
• For RAG systems, log retrieved documents alongside the query so the context of each response can be reconstructed for compliance review.

Related concepts

• Prompt injection — the leading LLM security threat, listed first in the OWASP LLM Top 10
• AI agent security — LLM security extended to agentic systems with real-world capability
• Agentic AI risk — the threat category that arises when LLMs control tools and take autonomous action
• AI governance — the policy framework within which LLM security controls are defined and enforced