AI Model Evaluation Metrics: Accuracy, ASR, Refusal, FPR

To measure AI model accuracy, run the model on a held-out labeled test set and divide correct outputs by total outputs. Report precision, recall, and F1 beside it, since accuracy collapses on imbalanced data. For generative models, a rule decides “correct”: exact match, unit tests (pass@k), or a judge model. Security testing adds attack success and benign refusal rates.

Security evaluation asks two questions that quality metrics never ask: how often does an attack get through, and how often does the defense get in the way of legitimate use? Answering them takes a short list of rates, each with a numerator, a denominator, and a judge that decides which cases count. Change any of the three and the number changes with it, which is why two published figures with the same name are so often incomparable.

How to measure AI model accuracy and performance

Accuracy is the share of test cases a model gets right, and every classification rate comes from the same confusion matrix of true and false positives and negatives:

accuracy  = (TP + TN) / (TP + TN + FP + FN)
precision = TP / (TP + FP)
recall    = TP / (TP + FN)
F1        = 2TP / (2TP + FP + FN)

1. Hold out a labeled test set the model never saw in training or tuning, drawn from the inputs it will actually face.
2. Write the correctness rule first: label match for a classifier, normalized exact match for short answers, unit tests for code, a rubric for open-ended text.
3. Check the class mix. On data that is 99 percent negative, always predicting negative scores 99 percent accuracy, the trap Google’s ML Crash Course cites against accuracy on imbalanced sets; scikit-learn’s balanced accuracy, the mean of per-class recall, avoids it.
4. Sweep the threshold when the model outputs a score, and report ROC AUC with the operating point you would deploy, plus an interval and repetition count (below).

AI model metrics by task type

Pick the row for the model’s output, and add the last row whenever input can be hostile, with test sets built as in how to benchmark LLM security.

Foundation model evaluation and GPT benchmark scores

A foundation model is scored on many benchmarks under one fixed prompting protocol, since no single task covers a general-purpose model. HELM set the pattern: 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) over 16 core scenarios for 30 models.

OpenAI open-sourced simple-evals to be transparent about the accuracy numbers published with its models: MMLU, MATH, GPQA, DROP, MGSM, HumanEval, SimpleQA, BrowseComp, and HealthBench, run zero-shot with chain-of-thought. It stopped updating for new models in July 2025, so quote any GPT figure with its date and protocol.

Code is scored by execution. HumanEval has 164 problems averaging 7.7 unit tests each, and pass@k counts a problem solved if any of k samples passes. Codex solved 28.8 percent with one sample and 70.2 percent with 100, the any-success-in-N effect described below for attacks.

Scores also wear out. MMLU-Pro widened answer choices from four to ten, and accuracy fell 16 to 33 percent against MMLU. METR’s time horizon, the human-expert task length a model completes with 50 percent probability, has no fixed ceiling; it doubled about every seven months over six years in METR’s March 2025 analysis.

Why quality metrics do not transfer

Every row of the task table except the last measures how good or fluent an output is when the input is cooperative. A security test asks something else. The input is adversarial, and the question is not whether the output resembles a reference but whether it violates a policy, leaks data, or triggers an action the user never asked for. A model can post strong numbers on every quality metric and still comply with a jailbreak, so quality scores belong in a security report only as context. Why a capability score is the wrong basis for choosing a model for an exposed deployment is covered in choosing an LLM on security evidence, not MMLU scores.

Attack success rate

Attack success rate (ASR) is the share of attack attempts that produced the behavior the attacker was after:

ASR = successful attempts / total attempts

Every term in that ratio is a decision, and each one has to be written down before the run:

• Attempt unit. One prompt, one multi-turn conversation, or one harmful behavior tried with several attack templates. When every behavior gets the same set of templates, a per-behavior rate (a behavior counts as broken if any template succeeds) can never be lower than the per-prompt rate on the same data, so the two are not interchangeable.
• Success criterion. What the judge has to see for an attempt to count. A response that refuses and then explains the general method anyway is a partial success, and the rule for handling it moves the number.
• Denominator. Which behaviors are in scope. Padding the corpus with requests every model refuses lowers ASR without making anything safer.

Then stratify. An ASR averaged across categories hides the tier that matters, so report it per behavior category, per technique family, and per severity tier, and keep static-corpus ASR apart from adaptive ASR, where attacks are rewritten against the specific target. The full protocol is in benchmarking jailbreak resistance.

Repetition: per attempt or any success in N

Model outputs are sampled, so one attempt per payload measures luck, and repeating attempts changes what the number means. If a single attempt succeeds with probability p and attempts are independent, the chance that at least one of N attempts succeeds is:

P(at least one success in N) = 1 - (1 - p)^N

At p = 2 percent, that is about 18 percent over 10 attempts and about 64 percent over 50. Real attempts are not fully independent, so read this as the direction of the effect rather than a forecast. The reporting rule follows directly: state the repetition count next to every rate, and say whether the rate is per attempt or any success in N. A per-attempt figure from one source and an any-success figure from another are different quantities wearing the same label. Fixing the count before the run is part of the AI security testing method.

Benign refusal rate

ASR on its own rewards a model that refuses everything. The counterweight is the benign refusal rate:

benign refusal rate = benign requests refused / benign requests

Two details decide whether it means anything. First, the benign set has to sit near the policy boundary: security research questions, medical and legal questions asked in good faith, fiction with conflict in it. Refusal measured on obviously harmless requests is near zero for every model and separates nothing. Second, keep it apart from refusal on the attack corpus. Refusing an attack is the outcome you want and is already inside the ASR calculation; refusing a legitimate request is a cost. A single “refusal rate” that mixes the two hides both.

ASR and benign refusal rate, measured at the same fixed configuration, give the operating point. Two models at 6 percent ASR, one refusing 4 percent of boundary requests and the other 22 percent, have identical attack robustness and are different products. The paired method is in red-team eval methodology: ASR and refusal rate.

Detector metrics: TPR, FPR, and precision at your base rate

When the thing under test is a guardrail or classifier rather than the model, the vocabulary becomes a confusion matrix over an attack corpus and a benign corpus:

• True-positive rate (recall, detection rate) = attacks flagged / attacks
• False-positive rate = benign inputs flagged / benign inputs
• Precision = flagged inputs that were attacks / all flagged inputs

TPR and FPR describe the detector at a given threshold. Precision does not: it depends on how often attacks occur in the traffic. Suppose 1 request in 1,000 is an attack, and a detector catches 95 percent of attacks at a 1 percent false-positive rate. Per 1,000 requests it flags about 0.95 real attacks and about 10 benign ones, so fewer than 1 alert in 10 is real (precision near 9 percent). Run the same detector on a test set that is half attacks and precision comes out near 99 percent. Neither figure is wrong; only one describes production.

The same base-rate effect is why accuracy and F1 mislead here. At 1 attack in 1,000 requests, a detector that flags nothing is 99.9 percent accurate. F1 is built on precision, so it inherits the dependence on the test mix: an F1 from a balanced benchmark does not carry over to skewed production traffic. Report TPR and FPR with the threshold stated, and when you quote precision, state the attack prevalence it assumes.

Because most detectors expose a score, the honest presentation is a curve rather than a point: sweep the threshold, plot TPR against FPR, and name the operating point you would actually deploy, such as the highest TPR at or below 1 percent FPR. The corpus design behind both axes, with a benign set that is deliberately injection-adjacent and an attack set tiered from direct to adaptive, is in how to benchmark a prompt-injection detector. Published clean-data and adversarial scores for named guardrails, with their sources, are in the AI guardrail tools comparison.

Model-based evaluation and judge agreement

Model-based evaluation, where a second model grades the output, fills in wherever no exact reference exists. In the MT-Bench and Chatbot Arena study, GPT-4 judges matched human preferences over 80 percent of the time, the same level humans reach with each other, while showing position, verbosity, and self-enhancement biases. Inspect, from the UK AI Security Institute and Meridian Labs, ships model_graded_qa() and model_graded_fact() scorers next to exact() and f1().

ASR and refusal rate are both decided by a judge: a keyword rule, a fine-tuned classifier, or an LLM-as-a-judge. The judge is a classifier with error rates of its own, so it needs its own metric. Hand-label a sample of responses from both corpora and report how often the judge agrees with the human labels. LLM judges carry the biases above, verbosity bias (favoring longer, confident-sounding answers) among them, so pin the judge’s version exactly as you pin the target’s.

Raw agreement can flatter a judge. If 90 percent of responses in an attack run are refusals, a judge that labels everything a refusal agrees with the humans 90 percent of the time while catching none of the successes. Cohen’s kappa corrects for the agreement expected by chance:

kappa = (observed agreement - chance agreement) / (1 - chance agreement)

In that example, chance agreement is also 90 percent, so kappa is 0. Report kappa or per-class agreement next to the raw figure, and report it separately for the attack and benign sets, because a judge that spots harmful completions well can still misread a hedged refusal.

Known judge error can also be corrected for. If the judge catches a fraction s of real successes and wrongly flags a fraction f of non-successes, a measured ASR of m implies a true rate of (m - f) / (s - f). A measured 12 percent with s = 0.90 and f = 0.02 works out to about 11.4 percent. The correction is only as good as the labeled sample behind s and f, which is one more reason to keep raw responses so a better judge can re-score them later, as the reproducible eval harness design recommends.

Sample size: every rate needs an interval

A rate from a few hundred attempts is an estimate, and its margin is often wider than the gap between the systems being compared. For a 10 percent ASR measured on 200 attempts, a 95 percent Wilson interval runs from about 6.6 to 14.9 percent; a second system at 13 percent on its own 200 attempts is well inside the noise. Zero successes does not mean zero risk either: with 0 of 200, the 95 percent upper bound is about 3/200, or 1.5 percent.

Non-determinism adds a second layer on top of sampling error. Re-running the same configuration produces a spread, so report the mean and range over at least five runs, as the methodology posts on this site require, and print the attempt count next to every rate.

Agents: targeted attack success and utility under attack

For tool-using agents, the ASR and benign-refusal pair becomes targeted attack success rate (how often an injected instruction got the agent to carry out the attacker’s action) and benign utility under attack (how often the agent still finished the user’s real task with the injection present). A defense that refuses to act on any external content drives the first toward zero and wrecks the second, so the two are only meaningful together. The benchmarks that measure this pair are covered in measuring prompt-injection robustness in agents.

Overhead: added latency and cost

A defense also has an operating cost that decides whether it ships: added p95 latency, measured at the tail rather than the median, and cost per 1,000 checks, including any model the defense itself calls. A filter with an excellent TPR that adds seconds to every request, or costs more per call than the model it protects, gets switched off. The Scanner Tradeoff Explorer shows how detection, false-positive rate, latency, and cost pull against each other on illustrative curves, not measurements.

The metric card

Where the frameworks stop

The NIST AI Risk Management Framework’s Measure function calls for organizations to monitor AI system performance and trustworthiness continuously, including anomaly detection and audit-trail transparency for model changes. It says what to measure and govern, but it does not fix a benchmark set or a metric definition, so comparability across teams is still something each team has to construct, by publishing definitions like the ones above next to every number. neuralwatch.org tracks regulatory developments around AI RMF implementation and EU AI Act measurement requirements.

Where this fits in the network

aisecbench.com computes none of these metrics itself. It collects results other people published, and the definitions above are what a published number has to state before it can be compared with another. For the suites that supply the corpora, see the reference to published AI security benchmark suites. For the attack techniques behind those corpora, aisec.blog covers the offensive side; for applying these rates to live inference traffic rather than held-out test sets, sentryml.com covers ML observability tooling; and for model-quality evaluation outside security, see llmops.report.

What makes a security metric unusable

• A rate with no denominator, attempt unit, or repetition count
• ASR without a benign refusal rate measured at the same configuration
• One “refusal rate” that mixes refused attacks with refused legitimate requests
• Detector accuracy or F1 from a balanced test set presented as production performance
• Precision quoted without the attack prevalence it assumes
• Any judge-scored number without judge agreement, reported per corpus

Every metric here is a ratio, and a ratio is only as honest as its definitions. Publish the numerator, the denominator, the judge, and the interval with every figure, and two security evaluations become something a reader can actually compare.

FAQ

what is the difference between ai model evaluation and ai security evaluation

Model evaluation asks whether outputs are correct on cooperative inputs, using accuracy, F1, pass@k, or benchmark scores. Security evaluation asks how often hostile inputs succeed and how often defenses block legitimate use, measured as attack success rate, benign refusal rate, and detector false-positive rate. A model can pass the first and fail the second.

what metrics assess ai capability maturity

For the models, track measures without a fixed ceiling, such as METR’s 50 percent time horizon, across generations. For an evaluation program, maturity means every number carries a frozen held-out test set, an interval, a judge with measured agreement, repeated runs, and an adaptive attack slice.

how do you evaluate an llm when there is no single correct answer

Write a rubric, have a judge model apply it, and hand-label a sample to measure how often the judge agrees with people. Report Cohen’s kappa rather than raw agreement, check for position and verbosity bias, and pin the judge model’s version so later scores stay comparable with earlier ones.