Benchmark Reference
AI Security Benchmarks: A Reference to 10 Test Suites
What AdvBench, HarmBench, JailbreakBench, AgentDojo, AgentHarm, Agent-SafetyBench, SEC-bench, CyberSecEval, P…
AI Sec Bench — Published benchmarks of AI security tools, collected and compared.
AI security tools, side by side.
Independent
Est. 2026
Est. 2026
Tool Comparisons
AI Firewall Placement: Lakera, NeMo, Bedrock, Azure
Where Lakera Guard, NeMo Guardrails, Bedrock Guardrails, and Azure AI Content Safety sit in the LLM request path, and what each placement costs.
Read the article →Latest reviews
Testing Guides
AI Security Testing: A Method for LLM and Agent Systems
AI security testing across four layers: how to scope an assessment, which published standards supply the test…
Testing Guides
How to Benchmark LLM Security: A Repeatable Method
Benchmark LLM security repeatably: define the threat model, pick suites that map to it, pin the target, and r…
Tool Comparisons
Open Source LLM Security Scanners: A Practitioner's Field Guide
Garak, NeMo Guardrails, PyRIT, and ARTKIT compared: how the leading open source LLM security scanners differ …
Model Evaluation
How AI Model Evaluation Metrics Work: A Practitioner's Guide
How AI model evaluation metrics work, from BLEU and ROUGE to BERTScore, perplexity, MMLU, and LLM-as-a-Judge:…
Tool Comparisons
The AI Security Tools Directory: 40+ Tools Compared (2026)
A maintained 2026 directory of 40+ AI and LLM security tools, comparing scanners, runtime guardrails, injecti…
Start here
The reference pages this site is built around, grouped by the question they answer.
Method
- AI security testing method
Scope an assessment across all four layers
- How to benchmark LLM security
Threat model, suite, pinned target, reported rate
- How to test AI agent security
Tool calls, memory, and inter-agent trust
Tool comparisons
- AI guardrail tools compared
Lakera, NeMo, Bedrock, Guardrails AI
- LLM red teaming tools compared
PyRIT, garak, Promptfoo and the test sets
- AI security tools directory
40+ scanners, guardrails, and monitors
Benchmark reference
- AI security benchmark suites
What each public corpus measures, and where it stops
- AdvBench vs HarmBench vs JailbreakBench
Why the three are not interchangeable
- Reproducible scanner benchmarks
Everything a published result has to pin
Or start from the numbers: the Scanner Tradeoff Explorer filters published detection, false-positive, latency, and cost figures against a budget you set.
Independent, specialist, and free to read
AI Sec Bench publishes focused, sourced guides on a single topic. No paywall, no account, no ad tracking.
Subscribe
AI Sec Bench — in your inbox
Published benchmarks of AI security tools, collected and compared — delivered when there's something worth your inbox.
No spam. Unsubscribe anytime.