Topics
Browse posts by category and tag — every topic we cover, with the latest pieces under each.
Tags
- #llm-security 9
- #prompt-injection 8
- #benchmark 7
- #methodology 7
- #red-teaming 6
- #evaluation 5
- #guardrails 3
- #jailbreak 3
- #advbench 2
- #ai-security 2
- #attack-success-rate 2
- #benchmarking 2
- #content-safety 2
- #harmbench 2
- #jailbreakbench 2
- #owasp-llm-top-10 2
- #red-team 2
- #reproducibility 2
- #security-testing 2
- #agent-security 1
- #agents 1
- #ai-agents 1
- #ai-firewall 1
- #ai-guardrails 1
- #ai-metrics 1
- #benchmarks 1
- #classifier 1
- #detection 1
- #eval 1
- #eval-harness 1
- #false-positive-rate 1
- #garak 1
- #jailbreak-detection 1
- #llm-benchmarks 1
- #llm-quality 1
- #llm-scanner 1
- #mlops 1
- #model-evaluation 1
- #model-quality 1
- #observability 1
- #open-source 1
- #production-llm 1
- #pyrit 1
- #refusal-rate 1
- #robustness 1
- #safety 1
- #tool-comparison 1
- #tools 1
Categories
Benchmark Methodology 7 posts
- Designing a Reproducible AI-Security Eval HarnessA reproducible AI-security evaluation is an engineering artifact, not a notebook. The harness design that separates corpus, target, judge, and report.
- Measuring Prompt-Injection Robustness in Tool-Using AgentsPrompt-injection robustness for an agent is not a single number — it is utility-under-attack against targeted attack success.
- Red-Team Eval Methodology: Attack Success Rate With Refusal RateAn LLM red-team evaluation that reports attack success rate without reporting refusal rate is half a measurement.
- Benchmarking LLM Jailbreak Resistance: Attack Success RateAttack success rate is the headline metric for jailbreak resistance, and almost everyone computes it in a way that isn't comparable across runs.
- Reproducible LLM Scanner Benchmarks: What Everyone Forgets to PinAn LLM security scanner benchmark that isn't pinned to a model version, a seed, and a corpus hash isn't reproducible.
- Benchmarking Jailbreak Classifiers: The Asymmetry Nobody ReportsJailbreak classifiers are graded on attack recall and almost never on the cost of being wrong. That asymmetry is the whole story. Here's how to measure it.
Tool Comparisons 5 posts
- AI Firewall Placement: Lakera, NeMo, Bedrock, AzureWhere Lakera Guard, NeMo Guardrails, Bedrock Guardrails, and Azure AI Content Safety sit in the LLM request path, and what each placement costs.
- Open Source LLM Security Scanners: A Practitioner's Field GuideGarak, NeMo Guardrails, PyRIT, and ARTKIT compared: how the leading open source LLM security scanners differ on coverage, fit, and maintenance.
- The AI Security Tools Directory: 40+ Tools Compared (2026)A maintained 2026 directory of 40+ AI and LLM security tools, comparing scanners, runtime guardrails, injection detection, and observability.
- Best AI Guardrail Tools 2026: Lakera vs NeMo vs BedrockA comparison of the leading AI guardrail tools in 2026, covering Lakera Guard, NVIDIA NeMo, AWS Bedrock Guardrails, and Guardrails AI on real trade-offs.
- Best LLM Red Teaming Tools 2026: A Practitioner's EvaluationA documentation-based comparison of the leading LLM red teaming tools in 2026: PyRIT, Garak, Promptfoo, and the HarmBench and JailbreakBench test sets.
Testing Guides 3 posts
- AI Security Testing: A Method for LLM and Agent SystemsAI security testing across four layers: how to scope an assessment, which published standards supply the test cases, which tools run them, and what to report.
- How to Benchmark LLM Security: A Repeatable MethodBenchmark LLM security repeatably: define the threat model, pick suites that map to it, pin the target, and report attack success rate with refusal rate.
- How to Test AI Agent Security: A Practical Evaluation GuideTesting AI agent security needs a different approach than static LLM red teaming. The attack surface, a test methodology, and the OWASP agentic checklist.
Benchmark Reference 2 posts
- AI Security Benchmarks: A Reference to 10 Test SuitesWhat AdvBench, HarmBench, JailbreakBench, AgentDojo, AgentHarm, Agent-SafetyBench, SEC-bench, CyberSecEval, PINT, and RAID each measure, and where each stops.
- AdvBench vs HarmBench vs JailbreakBench: Key DifferencesAdvBench, HarmBench, and JailbreakBench are not interchangeable, and treating them as one undermines every comparison built on top.
Model Evaluation 2 posts
- How AI Model Evaluation Metrics Work: A Practitioner's GuideHow AI model evaluation metrics work, from BLEU and ROUGE to BERTScore, perplexity, MMLU, and LLM-as-a-Judge: when each applies and where it fails.
- LLM Benchmark Fidelity: Why MMLU Won't Predict Production QualityModels with identical MMLU scores behave very differently in production. Where benchmark fidelity breaks down and what to measure instead before shipping.