A team choosing a model for production usually starts from published benchmark scores: MMLU 78 against MMLU 82, HumanEval 65 against 71. For a deployment that will face hostile input, those scores answer the wrong question. MMLU is a multiple-choice knowledge test across 57 subjects. Nothing in it is adversarial, and nothing in it measures whether a model refuses a harmful request, follows an instruction planted in a retrieved document, or turns away legitimate users. A model can win the capability comparison and lose the security one, and the capability score will not tell you which happened.
This post covers where capability benchmarks stop tracking security behavior, and the security evidence worth collecting before a model is chosen. General model-quality selection, for tasks with no adversary in the loop, is covered on llmops.report; the angle here is what changes when there is one.
Where capability benchmarks stop tracking security
1. Distribution mismatch
Capability suites sample cooperative questions. An exposed application’s traffic also contains deliberate attempts to subvert the model: jailbreak prompts, instructions hidden in documents and tool output, requests built to extract the system prompt. None of that distribution appears in a knowledge test, so a knowledge score cannot describe how the model handles it.
2. Contamination, on both sides
Public benchmarks have been in training data for years, so a high public score is partly a measure of memorization. Security corpora carry the same problem in a more dangerous form: a model or guard tuned on widely published jailbreak sets can post a low attack success rate on those sets without being robust to anything new. Hold out a private slice, and add attacks written fresh against each candidate.
3. Format
Capability benchmarks are mostly multiple-choice. Security failures show up in open-ended generation, in structured output that a downstream component parses or executes, and in tool calls. Picking A, B, C, or D well says nothing about whether a model’s output is safe to hand to the next system.
4. Single turn versus multi-turn
Most benchmarks are single-turn. Multi-turn escalation, where an innocuous opening establishes context that a later turn exploits, only shows up in a multi-turn test, and agents add tool-call sequences on top. A single-turn number cannot stand in for either.
5. Short context versus long context
Benchmarks typically use short prompts. Indirect prompt injection lives in long context: retrieved chunks, agent histories, pasted documents, tool responses. Test at the context lengths, and through the content sources, that the deployment will actually feed the model.
What to measure instead
Attack success with benign refusal, on a standardized set
Run each shortlisted model against a published, versioned behavior set and report attack success rate with its attempt unit, success rule, judge, and repetition count stated. Pair it with the refusal rate on benign requests that sit near the policy boundary, measured at the same configuration, because a model can buy a low attack success rate by refusing legitimate users. The definitions and arithmetic are in AI security evaluation metrics, the protocol in benchmarking jailbreak resistance, and the suites themselves in the reference to published AI security benchmark suites.
Injection robustness in your own configuration
The bare model is not what ships. Test each candidate behind your system prompt, with your retrieval path and tools connected, because injection succeeds or fails through those. For agents, report targeted attack success together with the agent’s benign utility under attack, as set out in measuring prompt-injection robustness in agents.
A scanner pass for breadth
Run a portion of the garak probe set against each candidate endpoint as part of selection. Pin the scanner version and probe set so the run can be repeated, and use it to surface categories of weakness, such as encoding tricks, prompt injection, and data leakage, that a narrow behavior set can miss.
A domain set of your own
A few hundred cases built from the deployment itself: attacks that name your real tools, document formats, and business rules, plus benign requests drawn from real or expected traffic. It is the expensive part, and the only part that tests the system you will actually run. Once built, it is reusable across every later model upgrade.
Latency and cost with the guardrail in the path
Benchmark scores are often computed with no latency budget at all. Production has one, and every guardrail adds to it. Measure p95 latency and cost per request with the defenses you will run switched on, at the concurrency you expect.
A selection protocol
For a team choosing a model for a feature that will face untrusted input:
- Shortlist on capability. Use public benchmarks and broad suites such as HELM and BIG-bench to find three to five candidates that can do the job at all.
- Pin each candidate. Dated model snapshot, system prompt, sampling settings, and guardrail configuration, recorded before the first test.
- Run a standardized jailbreak behavior set. Record attack success rate with attempt unit, success rule, and repetition count, stratified by category.
- Run the benign boundary set at the same configuration and record the refusal rate.
- Test injection through the real retrieval and tool path. For agents, record targeted attack success and utility under attack.
- Add an adaptive slice: attacks rewritten against each candidate, reported separately from the static result.
- Calibrate the judge on a hand-labeled sample and report its agreement rate.
- Measure latency and cost with the guardrail in the path.
- Decide on the operating point: attack success, benign refusal, latency, and cost read together, not the leaderboard rank.
Where this fits in the network
aisecbench.com runs none of these tests itself. It collects security results other people published and records the conditions behind each one, which is what a selection decision needs before it can lean on a published number. For the metric definitions behind the protocol, see AI security evaluation metrics; for the full assessment method around it, AI security testing: a method for LLM and agent systems.
For the engineering side of running the chosen model, see llmops.report on cost observability and mlmonitoring.report on drift detection.
What we don’t recommend
- Choosing a model for an exposed deployment on capability scores alone
- Trusting a vendor safety figure without the corpus, judge, and configuration behind it
- Testing the bare model, then shipping it behind a different system prompt, toolset, or guardrail
- Reading a near-zero attack success rate without the benign refusal rate next to it
- Treating a static-corpus result as the model’s robustness ceiling
A capability score tells you whether a model can do the job. Only security evidence, collected against your own configuration, tells you whether it can still do the job while someone is attacking it.
