Methodology
-
AI Model Evaluation Metrics: Accuracy, ASR, Refusal, FPR
How AI security evaluations are scored: attack success rate, benign refusal rate, detector TPR and FPR, and judge agreement, with the math behind each.
-
Designing a Reproducible AI-Security Eval Harness
A reproducible AI-security eval is an engineering artifact: a harness that separates corpus, target, judge, and report, plus what a scanner run must pin.
-
Measuring Prompt-Injection Robustness in Tool-Using Agents
Prompt-injection robustness for an agent is not a single number — it is utility-under-attack against targeted attack success.
-
AdvBench vs HarmBench vs JailbreakBench: Key Differences
AdvBench, HarmBench, and JailbreakBench are not interchangeable, and treating them as one undermines every comparison built on top.
-
Red-Team Eval Methodology: ASR and Refusal Rate
A paired red-team evaluation measures attack success and benign refusal on separate corpora, with a calibrated judge and fixed model configuration.
-
Benchmarking LLM Jailbreak Resistance: Attack Success Rate
Attack success rate is the headline metric for jailbreak resistance, and almost everyone computes it in a way that isn't comparable across runs.
-
How to Benchmark a Prompt-Injection Detector Honestly
Most prompt-injection detector benchmarks are broken before the first request. Here is a test design that produces a number you can actually trust.