The AI jailbreak benchmark leaderboard landscape does not produce one defensible procurement winner. For a named snapshot of frontier-model safeguards, start with FAR.AI; for a repeatable CI release gate, run JailbreakBench or HarmBench against the complete application and use StrongREJECT as a second judge. MLCommons is a useful independent cross-check, but its current public results are de-identified. A high rank should qualify a model for application-specific testing, not approve it for production.
AI jailbreak benchmark leaderboard: current results
As of September 6, 2026, the clearest narrowly scoped, named comparison is FAR.AI’s study of four flagship systems. Its metric is the estimated cost of discovering a universal jailbreak, defined as one prompt template that elicits operationally compliant responses for more than three-quarters of a domain’s harmful goals. The study used 360 goals across CBRNE and offensive-cyber domains and a taxonomy of 67 static techniques (FAR.AI technical report).
| Rank within FAR.AI’s study | System | Random-search result | Procurement interpretation |
|---|---|---|---|
| 1= | Claude Fable 5 | No universal jailbreak found | Right-censored result, not proof of resistance to every attack |
| 1= | GPT-5.6 Sol | No universal jailbreak found | Same limitation: the search ended without a qualifying result |
| 3 | Gemini 3.1 Pro | 18 universal jailbreaks; roughly $278 average discovery cost | Stronger than the fourth-ranked system under this harness |
| 4 | Grok 4.5 | 63 universal jailbreaks; roughly $58 average discovery cost | Lowest discovery cost and most random-search findings in this test |
This table is not a universal safety order. It covers four products, selected high-harm domains, and one compositional attack methodology. By contrast, MLCommons tested 39 text systems and found that none earned a jailbreak-resistance grade above “Good,” but de-identified every system in the v0.5 results. That supports a market-level conclusion, not a named vendor decision.
What it does: capabilities matrix
These are build-time evaluation resources, not runtime filters. “Output scanning” below means offline grading of a model response; it does not mean the suite can be inserted as a production content-safety classifier.
| Resource | Input scanning | Output scanning / jailbreak detection | PII redaction, hallucination check, RAG isolation | Audit logging / on-prem vs SaaS / language |
|---|---|---|---|---|
| FAR.AI Leaderboard | No; composes offline attack prompts | Three-stage evaluation of attack success | None | No runtime log; hosted research leaderboard, not a SaaS control; English study |
| JailbreakBench | No; evaluates submitted attacks and defenses | Llama-based jailbreak and refusal judges | None | Saved run metadata and artifacts; open-source local harness plus hosted leaderboard; English corpus |
| HarmBench | No; generates red-team test cases | Behavior-conditioned classifiers for text, contextual, and multimodal outputs | None | Local result files; open-source local pipeline; corpus-defined coverage |
| StrongREJECT | No; applies jailbreak transformations offline | Rubric-based or fine-tuned judge scores refusal, specificity, and harmful usefulness | No generic factuality or RAG check | Local result data; open-source local package; six harmful-content categories |
| MLCommons AILuminate | No; applies template and encoding attacks | Common evaluator compares baseline safety with under-attack safety | None | Hosted de-identified results plus local tooling; v0.5 is en-US and covers text and limited text-plus-image |
The missing columns matter. None of these resources is a PII redactor, secret scanner, output validator, retrieval-poisoning detector, or agent sandbox. They measure a control objective; they are not the control.
Where it fits
The benchmark should exercise the same path that will ship:
CI attack corpus -> input runtime filter -> system prompt + model + RAG context
-> tool allowlist / agent sandbox -> output validator
-> benchmark judge -> score, artifacts, human review -> release gateThe production path stops at the user after the output validator. The benchmark path forks responses to an offline judge and retains enough metadata to reproduce the run: model and guardrail versions, system prompt, decoding settings, retrieved context, tool policy, and judge version. Testing only the base model omits the semantic firewall, RAG isolation, and output controls that can change the application result.
Trade-offs that change the ranking
Judge error and false positives. StrongREJECT’s paper reports a Spearman rank correlation of 0.900 for its fine-tuned evaluator and 0.846 for its rubric evaluator against human judgments; their all-response mean absolute errors were 0.084 and 0.077, respectively. Those are evaluator-agreement results on the study dataset, not a production guardrail’s false-positive rate. A procurement bake-off still needs a benign, policy-boundary set and separate false-refusal reporting.
Latency and throughput. These suites run offline, so they add no production p95 latency unless an evaluator is repurposed as a runtime filter. The public leaderboard scores do not provide production p95 or throughput cost. FAR.AI’s cost-to-jailbreak estimates attacker discovery spend, not per-request inference cost. Runtime candidates should be load-tested at the expected context length and concurrency.
Integration effort. JailbreakBench is the lighter choice for reproducing an attack/defense leaderboard with archived artifacts. HarmBench offers a broader red-teaming pipeline, but model adapters, GPUs for local attacks or judges, and policy mapping add operational work. StrongREJECT is most useful as an evaluator cross-check because it penalizes vacuous or incoherent “successful” responses rather than treating non-refusal alone as success.
Residual risk. Jailbreak testing covers part of OWASP LLM01:2025 Prompt Injection. It does not establish resistance to indirect injection through retrieved documents, sensitive-information disclosure, improper output handling, excessive agency, or vector and embedding weaknesses. Pair the leaderboard with RAG poisoning tests, tool allowlists, JSON-schema enforcement, and incident monitoring. This follows the NIST AI RMF Measure function, which calls for documented, repeatable evaluation under conditions similar to deployment.
Who should pick which benchmark
Who should use them: frontier-model procurement teams can use FAR.AI to narrow a shortlist; AppSec and MLOps teams can combine JailbreakBench or HarmBench with StrongREJECT in CI; governance teams can use MLCommons’ baseline-versus-attack framing for repeatable evidence. The attack families behind these tests are catalogued at aisec.blog.
Who should not rely on them alone: RAG or agent owners facing indirect prompt injection, multilingual applications outside the released corpora, and buyers comparing production guardrails on false-positive rate or p95 latency. Those teams need application-shaped tests and runtime-control evidence; guardml.io covers the defensive guardrail layer separately.
Related across the network
- Adversarial Attacks on Vision-Language Models — adversarialml.dev
- Best AI Red Teaming Tools for ML Models in 2026 — adversarialml.dev
- Training Data Extraction from LLMs: Carlini’s Findings — adversarialml.dev
