#benchmark
-
AI Security Benchmarks: A Reference to 10 Test Suites
What AdvBench, HarmBench, JailbreakBench, AgentDojo, AgentHarm, Agent-SafetyBench, SEC-bench, CyberSecEval, PINT, and RAID each measure, and where each stops.
-
Measuring Prompt-Injection Robustness in Tool-Using Agents
Prompt-injection robustness for an agent is not a single number — it is utility-under-attack against targeted attack success.
-
AdvBench vs HarmBench vs JailbreakBench: Key Differences
AdvBench, HarmBench, and JailbreakBench are not interchangeable, and treating them as one undermines every comparison built on top.
-
Benchmarking LLM Jailbreak Resistance: Attack Success Rate
Attack success rate is the headline metric for jailbreak resistance, and almost everyone computes it in a way that isn't comparable across runs.
-
Reproducible LLM Scanner Benchmarks: What Everyone Forgets to Pin
An LLM security scanner benchmark that isn't pinned to a model version, a seed, and a corpus hash isn't reproducible.
-
Benchmarking Jailbreak Classifiers: The Asymmetry Nobody Reports
Jailbreak classifiers are graded on attack recall and almost never on the cost of being wrong. That asymmetry is the whole story. Here's how to measure it.
-
How to Benchmark a Prompt-Injection Detector Honestly
Most prompt-injection detector benchmarks are broken before the first request. Here is a test design that produces a number you can actually trust.