Evaluation
-
10 AI and LLM Security Benchmarks Compared
Compare 10 AI security benchmarks for LLMs, agents, detectors, and software-security tasks, including their scope, metrics, limits, and reporting traps.
-
AI Model Evaluation Metrics: Accuracy, ASR, Refusal, FPR
How AI security evaluations are scored: attack success rate, benign refusal rate, detector TPR and FPR, and judge agreement, with the math behind each.
-
LLM Red Teaming Tools Benchmarked: Garak, PyRIT, Promptfoo
How Garak, PyRIT, and Promptfoo compare for reproducible red-team benchmarks, with a capability matrix and roles for HarmBench and JailbreakBench.
-
Red-Team Eval Methodology: ASR and Refusal Rate
A paired red-team evaluation measures attack success and benign refusal on separate corpora, with a calibrated judge and fixed model configuration.
-
Benchmarking LLM Jailbreak Resistance: Attack Success Rate
Attack success rate is the headline metric for jailbreak resistance, and almost everyone computes it in a way that isn't comparable across runs.
-
Choosing an LLM on Security Evidence, Not MMLU Scores
MMLU-style capability scores do not measure how a model behaves under attack. What to measure before choosing an LLM for a security-sensitive deployment.