How to Benchmark LLM Security: A Repeatable Method

Teams asking how to benchmark LLM security usually want one number, and the honest answer is that no single number exists. What does exist is a repeatable method: define the threat model, pick test suites that map to it, run them against a pinned model and system configuration, and report attack success rates alongside refusal behavior. Done that way, a security benchmark becomes something you can compare across model versions and procurement candidates instead of a one-off demo. This guide covers the decisions that matter, the main open-source suites, and the limits you should write into any report.

Start with a threat model, not a tool

The most common benchmarking mistake is running a scanner before deciding what “insecure” means for your deployment. The OWASP Top 10 for LLM Applications 2025 is the practical starting point: it names the failure classes worth measuring, from LLM01 (prompt injection) and LLM02 (sensitive information disclosure) through LLM06 (excessive agency) and LLM07 (system prompt leakage). A customer-facing chatbot cares most about LLM01 and LLM02; an agent with tool access has to weight LLM06 heavily; a RAG pipeline adds LLM08 (vector and embedding weaknesses). Your benchmark should sample attacks in proportion to which of these apply.

For a more formal vocabulary, NIST AI 100-2 E2025, finalized in March 2025, provides a taxonomy of adversarial machine learning attacks across the AI lifecycle, covering evasion, poisoning, and privacy attacks along with their mitigations. It is useful less as a test plan than as a shared language: when a report says “indirect prompt injection via retrieved content,” the taxonomy lets two teams agree on what was and was not in scope.

One scoping decision matters more than any other: are you benchmarking the model or the system? A raw model score tells you what the base weights refuse. A system score includes your system prompt, guardrails, tool allowlists, and output filters. Both are worth having, because the gap between them is exactly the value your runtime controls add. Benchmarking only the guarded system hides how much you are leaning on a filter; benchmarking only the raw model overstates production risk. Coverage of the guardrail layer itself, and what each product actually filters, is a separate evaluation problem we track at https://guardml.io.

Choose your suites: static benchmarks and adaptive scanners

The open-source landscape splits into two families with different jobs.

Static benchmark suites run a fixed set of test cases and score outputs, which makes results comparable across models and over time. CyberSecEval, from Meta’s Purple Llama project, evaluates two things: a model’s propensity to suggest insecure code and its compliance when asked to help with cyberattacks. The original paper covered seven models across the Llama 2, Code Llama, and GPT families and reported that more capable models tended to suggest insecure code more often, a finding worth remembering when a vendor equates “newer” with “safer.” The suite has since grown through several releases in the PurpleLlama repository, adding agent-focused and defensive-capability benchmarks.

HarmBench targets a different question: how well a model resists active red teaming rather than one-shot harmful prompts. The paper standardizes evaluation across 510 harmful behaviors and compares 18 automated red-teaming methods against 33 target LLMs and defenses. Two of its findings shape how you should read any vendor safety claim: no attack or defense in the study was uniformly effective, and robustness did not track model size. If a benchmark report only shows results against one attack method, HarmBench is the evidence that the number does not generalize.

Adaptive scanners probe your actual endpoint instead of scoring a fixed dataset. garak, NVIDIA’s open-source LLM vulnerability scanner, ships dozens of probe modules covering prompt injection, DAN-style jailbreaks, encoding-based injection, adversarial suffixes, glitch tokens, data leakage, and malware generation requests, with paired detectors that judge each response and structured JSONL reports per run. Microsoft’s PyRIT sits in the same family as a framework for building your own probing workflows rather than a fixed probe set. Scanners are the right tool for regression-testing a deployed system; static suites are the right tool for comparing models. Most programs need one of each. For the attack techniques these tools automate, the offensive side is covered in depth at https://aisec.blog.

Run it so the numbers mean something

A benchmark run is an experiment, and the usual experimental hygiene applies.

Pin everything. Record the model identifier and version, system prompt, temperature, guardrail product and version, and the suite’s own commit hash. Hosted models change silently, so an undated score is close to worthless.

Report attack success rate per category, not a blended total. An aggregate “94% safe” hides a 40% failure rate on the one category that matters for your deployment. Break results out by OWASP class or by probe module.

Measure over-refusal in the same run. A model that refuses everything scores perfectly on attack resistance and is useless in production. Pair each safety score with a benign-task pass rate so you can see the trade-off you are actually buying. This is the LLM analog of a false-positive rate, and it is the number vendors are least eager to publish.

Re-run on every change. Model upgrades, system-prompt edits, and guardrail version bumps all shift results. Treat the benchmark as a CI gate with a stored baseline, not an annual audit artifact.

Limits and residual risk

Every score from these suites is a lower bound on risk, not an upper bound. Static test sets leak into training data over time, and a model can score well on published prompts while failing paraphrases of them. HarmBench’s comparison shows attack effectiveness varies widely by method, so resistance to the 18 methods tested says little about the 19th. Scanners like garak sample a fixed probe library; a clean run means the probes it ran did not succeed, not that the system is safe. And none of these suites measure your deployment’s non-model attack surface: the tool integrations, retrieval pipeline, and output handling where much of the real risk in agentic systems now lives.

The defensible claim after a good benchmark run is narrow: this configuration, on this date, resisted this documented attack set at this rate. That claim, tracked over time and re-earned on every change, is what benchmarking LLM security can actually deliver. Layer human red teaming on top for the attacks no suite has automated yet.

Two companion pieces extend this method. For the wider assessment that surrounds a benchmark run — scoping by layer, inventorying trust boundaries, and reporting rates rather than verdicts — see AI security testing: a method for LLM and agent systems. For picking the corpus itself, the reference to published AI security benchmark suites covers what each of the ten most-cited suites measures and where each one stops.

Related across the network

• How to Scan an LLM for Prompt Injection: Tools, Method, and Limits — bestllmscanners.com
• AI Defense Techniques for LLMs: A Practitioner’s Guide — aidefense.dev
• Best Prompt Injection Detection Tools in 2026: The Evidence — aidefense.dev
• Jailbreak Detection for LLMs Explained: How Runtime Filters Work — aidefense.dev
• How to Detect Jailbreak Prompts: A Practitioner’s Guide — aimoderationtools.com