Topics
Browse everything AI Sec Bench publishes by category and tag: every topic covered here, with the most recent guides listed under each one.
Tags
- #llm-security 13
- #benchmark 9
- #prompt-injection 9
- #methodology 7
- #red-teaming 7
- #evaluation 6
- #attack-success-rate 4
- #guardrails 4
- #owasp-llm-top-10 4
- #false-positive-rate 3
- #advbench 2
- #ai-security 2
- #benchmarking 2
- #content-safety 2
- #harmbench 2
- #jailbreak 2
- #jailbreakbench 2
- #red-team 2
- #refusal-rate 2
- #security-testing 2
- #adversarial-robustness 1
- #agent-security 1
- #agents 1
- #ai-agents 1
- #ai-firewall 1
- #ai-guardrails 1
- #ai-jailbreak 1
- #ai-security-tools 1
- #ai-text-detection 1
- #detection 1
- #eval-harness 1
- #garak 1
- #jailbreak-detection 1
- #leaderboard 1
- #model-evaluation 1
- #model-selection 1
- #observability 1
- #open-source 1
- #promptfoo 1
- #pyrit 1
- #raid 1
- #reproducibility 1
- #robustness 1
- #tool-comparison 1
- #tools 1
Categories
Benchmark Methodology 8 posts
- RAID Benchmark AI Detector Results: What the Scores MeanRAID scores AI-text detectors at a fixed 5% false-positive rate across 11 models and 11 attacks. What top scores mean and where they mislead.
- AI Model Evaluation Metrics: Accuracy, ASR, Refusal, FPRHow AI security evaluations are scored: attack success rate, benign refusal rate, detector TPR and FPR, and judge agreement, with the math behind each.
- Designing a Reproducible AI-Security Eval HarnessA reproducible AI-security eval is an engineering artifact: a harness that separates corpus, target, judge, and report, plus what a scanner run must pin.
- Measuring Prompt-Injection Robustness in Tool-Using AgentsPrompt-injection robustness for an agent is not a single number — it is utility-under-attack against targeted attack success.
- Red-Team Eval Methodology: ASR and Refusal RateA paired red-team evaluation measures attack success and benign refusal on separate corpora, with a calibrated judge and fixed model configuration.
- Benchmarking LLM Jailbreak Resistance: Attack Success RateAttack success rate is the headline metric for jailbreak resistance, and almost everyone computes it in a way that isn't comparable across runs.
Tool Comparisons 6 posts
- Prompt Injection Scanner Tools: Benchmarks & EvasionHow prompt injection scanners compare on published robustness and evasion benchmarks: cloud guardrails, Lakera Guard, Prompt Guard 2, and NeMo.
- AI Firewall Placement: Lakera, NeMo, Bedrock, AzureWhere Lakera Guard, NeMo Guardrails, Bedrock Guardrails, and Azure AI Content Safety sit in the LLM request path, and what each placement costs.
- Open Source LLM Security Scanners ComparedGarak, NeMo Guardrails, PyRIT, and ARTKIT compared: how the leading open source LLM security scanners differ on coverage, fit, and maintenance.
- 40 AI Security Tools Compared: 2026 DirectoryCompare 40 AI and LLM security tools by category, function, open-source status, and maturity across scanners, guardrails, detection, and monitoring.
- AI Guardrail Tools Benchmarked: Lakera, NeMo, BedrockAI guardrail tools benchmarked on adversarial robustness, latency, and cost: Lakera Guard, NVIDIA NeMo, AWS Bedrock Guardrails, and Guardrails AI.
- LLM Red Teaming Tools Benchmarked: Garak, PyRIT, PromptfooHow Garak, PyRIT, and Promptfoo compare for reproducible red-team benchmarks, with a capability matrix and roles for HarmBench and JailbreakBench.
Benchmark Reference 3 posts
- OWASP LLM Top 10 Benchmark Mapping: What Tests Each RiskWhich public benchmarks and scanner probes test each OWASP LLM Top 10 2025 risk, where coverage is thin, and why tool tags still use 2023 IDs.
- 10 AI and LLM Security Benchmarks ComparedCompare 10 AI security benchmarks for LLMs, agents, detectors, and software-security tasks, including their scope, metrics, limits, and reporting traps.
- AdvBench vs HarmBench vs JailbreakBench: Key DifferencesAdvBench, HarmBench, and JailbreakBench are not interchangeable, and treating them as one undermines every comparison built on top.
Testing Guides 3 posts
- AI Security Testing: A Method for LLM and Agent SystemsAI security testing spans application, model, infrastructure, and data risks, with repeatable cases, calibrated judging, metrics, and CI checks.
- How to Benchmark LLM Security: A Repeatable MethodBenchmark LLM security repeatably: define the threat model, pick suites that map to it, pin the target, and report attack success rate with refusal rate.
- AI Agent Security Testing: A Practical GuideTesting AI agent security needs a different approach than static LLM red teaming. The attack surface, a test methodology, and the OWASP agentic checklist.