AI security
A practical LLM red teaming guide for prompt injection, jailbreaks, data leakage, tool misuse, RAG attacks, agent safety, adversarial testing, evals, and remediation.
Comparison
| Option | Best for | Strengths | Tradeoffs | Use when |
|---|---|---|---|---|
| Manual red teaming | Novel workflows, human judgment, exploratory testing, and business-specific abuse cases | Finds creative failures that automated suites often miss. | Harder to scale and reproduce without strong documentation. | The product is new, high-impact, or has unusual tool and data flows. |
| Automated adversarial evals | Regression testing, prompt changes, model upgrades, and repeated security checks | Scales across releases and makes fixes measurable. | Can miss novel attacks and can create false confidence if cases are shallow. | Known risk categories need to be checked on every release. |
| External or specialist red team | High-risk, enterprise, regulated, or public-facing systems | Adds independence, domain expertise, and adversarial perspective. | Requires scope, safe testing rules, data handling, and remediation ownership. | The system can affect money, safety, privacy, reputation, or regulated workflows. |
LLM red teaming should test the full application: prompts, retrieved documents, tool permissions, memory, file uploads, output handling, external APIs, logging, and human handoff.
Use known taxonomies such as OWASP LLM risks, then add product-specific abuse cases. A support bot, coding agent, finance assistant, and voice agent should not share the exact same test plan.
A red-team finding is not closed when a prompt is edited. It is closed when the system has a durable control, a test case, an owner, and monitoring for recurrence.
Decision Rules
Red-team the full LLM application, not only the base model.
Use manual testing for novel high-risk workflows and automated evals for regression.
Escalate external red teaming for public, regulated, or high-impact AI systems.
Convert every meaningful finding into a test, owner, and remediation record.
Related Guides
Turn red-team findings into layered controls.
OpenAutomate regression tests after red-team findings.
OpenBuild layered controls after red-team findings.
OpenTest private knowledge systems against leakage and injection.
OpenAutomate red-team regression tests.
OpenChinese Archive
Topic Hubs
Topic hub
Plan RAG systems, local LLM deployment, model APIs, cloud AI platforms, vector databases, evaluation, observability, rate limits, and cost optimization.
OpenTopic hub
Compare AI security controls, governance frameworks, compliance automation, data privacy, vendor questionnaires, red teaming, SIEM, SOAR, XDR, CNAPP, DSPM, DLP, PAM, GRC, and risk tooling.
OpenIndustry Pages
Industry page
Compare AI cybersecurity tools for SOC analysts, SIEM, SOAR, XDR, CNAPP, exposure management, DSPM, DLP, PAM, email security, AI GRC, vendor risk, red teaming, and compliance automation.
OpenIndustry page
Compare AI tools for ITSM, AIOps, SaaS management, LLM observability, gateways, rate limits, fallback routing, enterprise search, knowledge management, and IT governance.
OpenFAQ
It is adversarial testing of an LLM application to find safety, security, privacy, tool-use, retrieval, and abuse failures before attackers or users find them.
No. It provides evidence and reveals failures, but it cannot prove absence of risk. It should be paired with guardrails, evals, monitoring, human review, and incident response.
Both. Do pre-launch testing for known risks, then keep adversarial evals and monitoring running as models, prompts, tools, and retrieval data change.
Source Links
Reference
Official OpenAI safety guidance including adversarial testing and human oversight.
OpenReference
Microsoft AI Red Team guidance and resources.
OpenReference
Microsoft open-source framework for generative AI red teaming.
OpenReference
OWASP risk taxonomy for LLM applications.
OpenThe strongest decision is always local to your workflow. Save the vendor links, define a representative task, record the exact prompt or command, and compare the final evidence instead of the marketing claim.
Return to the AI learning map