AI Security Testing: A Method for LLM and Agent Systems

AI security testing is the discipline of proving, with evidence, that a model-backed system behaves safely when someone attacks it. It is not the same activity as an application penetration test, and it is not the same activity as a model quality evaluation, though it borrows machinery from both. The difference is that the thing under test is probabilistic: the same payload can fail nine times and succeed on the tenth, and a pass on a single run means almost nothing.

That single property drives everything below. A test plan that produces a pass/fail verdict per payload will mislead you. A test plan that produces a rate, with a stated sample size and a pinned model version, will not.

This guide covers how to scope an AI security assessment, which published standards supply the actual test cases, which tools execute them, and which numbers belong in the report.

What AI security testing covers that appsec testing does not

A conventional web assessment assumes deterministic behaviour: a payload either triggers the bug or it does not. Model-backed systems break that assumption in four ways.

• Non-determinism. Temperature, sampling, and provider-side model updates all move the result. A refusal today is not a refusal next month.
• The instruction/data collapse. Everything the model reads is potentially an instruction. Retrieved documents, tool outputs, file names, and email bodies all sit in the same context window as the system prompt, which is the root cause behind most prompt-injection findings.
• A moving trust boundary. In an agent, the boundary is redrawn on every tool call. The relevant question is not “is this input trusted” but “what can this input reach right now”.
• Semantic failure. A model can produce a fully valid HTTP 200 response that is also a policy violation. There is no stack trace, so detection is itself a measurement problem with its own error rate.

Everything conventional still applies underneath: the inference endpoint has authentication, the vector store has access control, the model weights have a supply chain. AI security testing adds a layer, it does not replace one.

Scope the assessment by layer, not by feature

The OWASP AI Testing Guide, released as version 1 in late 2025, structures assessments across four layers: application, model, infrastructure, and data. Scoping by layer is the fastest way to find the parts of a system nobody has tested, because teams reliably test the application layer and reliably skip the other three.

Layer	What is under test	Representative test cases
Application	Prompt assembly, output handling, tool wiring, session state	Direct and indirect injection, output-handling flaws such as rendered markup or executed code, cross-session leakage
Model	The model itself, plus any fine-tune or adapter	Jailbreak resistance, refusal behaviour, harmful-capability elicitation, training-data extraction
Infrastructure	Serving stack, inference API, sandboxes, orchestration	Authentication and quota enforcement, sandbox escape from generated code, denial of wallet, model-file deserialisation
Data	Training sets, RAG corpora, memory stores, embeddings	Poisoning of ingested content, unauthorised retrieval across tenants, embedding-inversion exposure

The infrastructure and data rows are where the expensive findings live. An injection that changes a chatbot’s tone is a defect; an injection that reaches a tool with write access to a shared corpus is an incident that persists after the session ends.

Standards supply the test cases, so use them

Writing test cases from imagination produces uneven coverage. Four public documents cover the field between them, and each answers a different question.

Document	Answers	Use it for
OWASP Top 10 for LLM Applications 2025	What commonly goes wrong in LLM apps	The baseline checklist for any text-generation feature
OWASP Top 10 for Agentic Applications	What goes wrong when the model acts	Agent scoping: goal hijack, tool misuse, memory poisoning, inter-agent trust
NIST AI 100-2 E2025	The formal attack taxonomy and vocabulary	Naming findings consistently; mapping predictive-AI attacks as well as generative ones
MITRE ATLAS	How real attacks chained together	Building end-to-end scenarios rather than isolated payloads

The mechanical step that most teams skip: for every entry in the checklist you adopt, write down the test case that exercises it and the observable that decides the outcome. Categories with a design intent but no executable test are exactly where undetected exposure accumulates. The agent-specific half of that exercise is worked through in detail in the guide to testing AI agent security.

A test plan that survives contact with a real system

1. Inventory the trust boundary crossings

List every point where content the system did not author enters the context window, and every point where model output leaves as an action. Retrieval sources, uploaded files, tool responses, sub-agent messages, memory reads. This list is the scope document; anything not on it will not be tested.

2. Pin the target before you generate a single payload

Record the model identifier and version, the exact system prompt, temperature and sampling settings, the tool schema, and the guardrail configuration with its thresholds. An unpinned target makes a result unreproducible the moment the provider ships an update, which is the single most common reason two teams cannot replicate each other’s numbers. The full pinning discipline is set out in what a reproducible scanner benchmark has to record.

3. Assemble the corpus from published suites plus system-specific payloads

Public suites give you comparability; bespoke payloads give you relevance. Neither alone is sufficient. Which suite fits which question is covered in the reference to published AI security benchmark suites, and the differences between the jailbreak sets are unpacked in AdvBench, HarmBench, and JailbreakBench compared.

The system-specific half is what a generic suite cannot supply: your tool names, your document formats, your tenant identifiers, your business rules. An injection payload that names a real tool in the schema is far more likely to succeed than a generic one.

4. Run every case repeatedly

One attempt per payload measures luck. The Best-of-N Jailbreaking work quantified how badly: resampling the same prompt with trivial augmentations such as random capitalisation and character shuffling drove attack success to 89 percent on GPT-4o and 78 percent on Claude 3.5 Sonnet at 10,000 samples, and the authors report that success rate follows power-law-like behaviour in the sample count across text, vision, and audio. A single-attempt number is therefore a point on a curve, not a property of the system.

The practical consequence is that the repetition count is part of the metric, not a detail of the run. Fix it before the run, apply it uniformly across every payload, and print it next to every rate. Nobody outside a research budget runs 10,000 attempts per case, but a single-digit count with no stated denominator is not a measurement at all.

5. Grade with a judge whose error rate you know

Deciding whether an output constitutes a successful attack is a classification task, and the classifier has a false-positive and false-negative rate of its own. A keyword matcher for refusal phrases will score a polite non-answer as a success. A model-based judge will drift when its own version changes. Whichever you use, hand-label a sample of a few hundred outcomes and publish the judge’s agreement rate, otherwise every downstream number inherits an unknown error bar.

6. Test the defence separately from the model

Guardrails are a second system with their own failure profile. Measure detection rate, false-positive rate on genuinely benign traffic, added p95 latency, and cost per thousand calls. A filter that catches 99 percent of attacks while flagging 20 percent of legitimate requests will be switched off within a week of launch, which makes it worth zero regardless of its detection score. The detector benchmarking method covers corpus construction for exactly this, and the interactive tool explorer lets you filter published guardrail operating points against a latency, cost, and false-positive budget.

7. Report rates, never verdicts

A finding that says “the system is vulnerable to prompt injection” is not actionable. A finding that says “targeted tool-call hijack succeeded on 34 of 200 attempts against model X at temperature 0.7 with guardrail Y at threshold 0.8” can be retested after a fix and compared against the next model version. Every number in the report carries its denominator, its target version, and its judge.

The metric set worth reporting

Metric	Definition	Why it is required
Attack success rate	Successful attempts divided by total attempts	The headline number, meaningless without the denominator
Refusal rate	Attempts the target explicitly declined	Separates a hardened model from an evasive one; see ASR reported with refusal rate
False-positive rate	Benign traffic flagged by the defence	Determines whether the defence survives production
p95 latency added	Milliseconds the defence adds at the tail	Determines whether the defence ships at all
Judge agreement	Judge versus human labels on a sample	The error bar on every other row

Reporting attack success rate alone is the most common defect in published AI security results, and it is the reason two vendor claims about the same model are so often incomparable. The mechanics of computing a comparable rate are covered in benchmarking jailbreak resistance.

Where AI security tests go wrong

• Testing the model instead of the system. Provider-level safety training is not your control. The finding that matters is what the model can reach through your tools, not what it will say in isolation.
• Single-turn only. Multi-turn escalation, where an innocuous opening turn establishes context that a later turn exploits, is under-tested relative to how often it works.
• Corpus contamination. Public jailbreak sets appear in training data. A low success rate on a widely published corpus may measure memorisation rather than robustness.
• Testing the happy-path deployment. Assessments run against the staging configuration with debug tools enabled, or against the production configuration with the eval harness bypassing the guardrail, both produce numbers that describe a system nobody ships.
• One-shot assessment. A point-in-time result decays as soon as the provider updates the model. Continuous execution is the only form of this testing that stays true.

Putting it in CI

The subset of this work that belongs in continuous integration is narrow and worth getting right: a fixed corpus of a few hundred payloads, a pinned model version, a fixed repetition count, and a threshold that fails the build when the attack success rate regresses beyond a stated margin. Everything else, in particular adaptive and exploratory red teaming, stays a scheduled human-driven exercise, because an adaptive attacker is precisely what a fixed regression suite cannot simulate.

Harness structure matters more than tool choice here: keep corpus, target, judge, and report as four separately versioned components so that changing one does not silently invalidate history. That separation is the subject of the reproducible eval harness design.

Where to start

If none of this exists yet, the first useful week looks like this: inventory the trust boundaries, pin the target configuration, run a public jailbreak suite and a public injection suite at a fixed repetition count, hand-label two hundred outcomes to calibrate the judge, and publish the four rates above with their conditions. That is a baseline. Everything after it is comparison against that baseline.

For tool selection, the open source LLM security scanner field guide covers what garak, PyRIT, and their peers each cover, the red teaming tool comparison weighs them against each other, and the AI security tools directory maps the wider commercial landscape. For the benchmarking method underneath all of it, start with how to benchmark LLM security.