A RAID benchmark AI detector score is the share of machine-generated text a detector catches while misflagging only 5% of human text. The best results published by RAID’s organizers sit above 99% on clean text and at 97.7% under adversarial attack, but those detectors were trained on RAID’s own domains and models, so treat any RAID number as an upper bound.
The verdict for a security team: RAID, which its authors describe as the largest and most challenging benchmark of its kind, is the right place to build a shortlist. It is not evidence that a detector will hold up on your traffic, on newer generators, or against an attacker who knows which product you run. That gap matters for phishing triage, fake-review pipelines and fraud queues. NIST AI 100-4 treats detection as one transparency technique alongside provenance tracking and watermarking, not a standalone control.
What does the RAID benchmark test?
RAID tests whether a detector can separate human from machine text when the generator, domain, decoding settings and adversarial edits all vary at once. The ACL 2024 paper by Dugan et al. built 6,287,820 texts, attacked versions included, against 14,971 human-written documents, spanning 11 models, 8 domains, 4 decoding strategies and 11 attacks.
| Axis | What RAID varies | Why a defender cares |
|---|---|---|
| Generators | GPT-2 XL, GPT-3 (text-davinci-003), ChatGPT, GPT-4, Llama 2 70B Chat, Cohere and Cohere Chat, MPT-30B and its chat variant, Mistral 7B and its chat variant | Detectors do best on models they saw in training |
| Domains | arXiv abstracts, recipes, Reddit posts, book summaries, NYT news, poetry, IMDb reviews, Wikipedia | False-positive rates diverge by content type |
| Decoding | Greedy and sampling (T=1), each with and without a 1.2 repetition penalty | An attacker controls these settings at no cost |
| Attacks | Article deletion, homoglyph, number swap, paraphrase, synonym swap, misspelling, whitespace, upper-lower swap, zero-width space, paragraph insertion, alternative spelling | Cheap, scriptable edits |
The project repository has since grown the corpus past 10 million documents with an “extra” split of Python code, Czech news and German news. The labeled train split is 802 MB without adversarial text and 11.8 GB with it. Unlike the suites in our AI security benchmarks roundup, RAID measures a content classifier, not a model’s resistance to misuse.
How is a RAID score calculated?
A RAID score is accuracy on machine-generated text at a threshold tuned so the detector flags 5% of human-written text. Each detector gets its own threshold. The headline is a true-positive rate at a fixed false-positive rate, not raw accuracy or F1, and it is comparable only to other RAID scores at the same FPR.
The authors explain the choice in the paper’s appendix: the corpus holds roughly 40 machine texts per human text, so precision and F1 would flatter every detector, and a fixed FPR reflects how detectors get used when a false accusation is costly. If no threshold reaches the target FPR, RAID records that. In the original evaluation ZeroGPT bottomed out at 16.9% FPR. For the broader metric trade-offs, see how AI model evaluation metrics work.
The calibration step is the part buyers skip. At a naive 0.25 threshold the paper reports GLTR with a 100% FPR and LLMDet at 97.9%, and LLMDet still misflags 75.3% of human text at 0.95. Even with aggregate FPR pinned at 5%, per-domain rates drift: Originality reached 13% on Wikipedia, and Winston 13% on books and on recipes. A global threshold can be fine on average and unacceptable for the one content type you actually screen.
Mechanically, 10% of the data is held out as an unlabeled test split. Submitters run their detector over it, open a pull request with predictions and metadata, and a GitHub bot scores the entry.
What did the original RAID evaluation find?
The original evaluation found that current detectors are easily fooled by adversarial attacks, changes in sampling, repetition penalties and unseen generators. The authors concluded detectors are “not yet robust enough for widespread deployment or high-stakes use.” The field covered four neural classifiers (three RoBERTa variants and RADAR), four metric-based methods (GLTR, Binoculars, Fast-DetectGPT, LLMDet) and four commercial APIs (GPTZero, Originality, Winston, ZeroGPT).
Three results matter most for threat modeling:
- Decoding is an attack surface. Adding a repetition penalty cut accuracy by up to 32 points regardless of decoding strategy, and sampled text was harder to catch than greedy output. Binoculars scored 99.9% in one non-adversarial setting and 0.6% in another.
- Character-level attacks are the cheapest evasion. Under homoglyph substitution, Binoculars fell from 79.6% to 37.7% and Originality from 85.0% to 9.3%. GPTZero lost only 0.3 points, while the other five detectors in that comparison dropped 40.6 points on average. Synonym swap took Binoculars to 43.5%.
- Attack effects are detector-specific. Paraphrasing raised Originality to 96.7% and lifted the RoBERTa GPT-2 detector by 16.2 points. An aggregate score hides which evasion works against the tool you deploy.
The authors’ ethics statement also opposes using detectors in disciplinary or punitive settings.
Who tops the RAID leaderboard?
On the organizers’ published record, Leidos and Pangram lead. In the COLING 2025 shared task built on RAID, Leidos scored 99.4% and Pangram 99.3% at 5% FPR without attacks, and both reached 97.7% with all 11 attacks. Nine teams submitted 23 detectors. The live board at raid-bench.xyz changes as new entries arrive.
Read those numbers with the task design in view. Every domain and generator in the shared-task test set also appeared in training; only individual articles were held out. The organizers state plainly that “detectors doing well in constrained settings does not imply generalization to unseen models.” Averaged across submissions, the hardest attacks were homoglyph (49.2%), paraphrase (60.6%) and synonym swap (61.3%). Leidos won with four fine-tuned Distil-RoBERTa-Base classifiers, a reminder that a fixed distribution rewards fitting it.
The main leaderboard is split into two sections, one for detectors that self-report training on RAID and one for those that do not. That split is the most useful filter a buyer has. The Desklib model card, for example, claims top performance and lists the RAID dataset as training data, which makes its score an in-distribution result. That is legitimate in the right section, but it does not support a “most accurate” claim on unseen text. The RAID paper itself warns that repeated evaluation on a fixed set “will lead to overfitting, even if detectors are not explicitly trained on examples.”
How should a security team use a RAID score?
Use a RAID score to cut a long list to a short one, then re-measure on your own data before any production decision. The steps below are the minimum:
- Confirm the leaderboard section. A trained-on-RAID score measures fit to RAID’s distribution, not generalization.
- Re-calibrate on your own human text. The RAID authors recommend tuning thresholds on in-domain human data and reporting FPR. Pull a representative sample of known-human text from the channel you screen and set the threshold at the FPR your review queue can absorb.
- Read the per-attack breakdown. Homoglyph and zero-width-space attacks take a few lines of script. Unicode normalization and invisible-character stripping ahead of the detector are worth testing; measure accuracy with and without them.
- Check generator coverage. RAID’s newest generators are GPT-4, Llama 2 and Mistral 7B, and the authors note that generated-text datasets “naturally obsolesce.” For text from current frontier or fine-tuned open-weight models, RAID offers limited evidence.
- Check language coverage. The main splits are English; non-English text appears only as Czech and German news in the extra split.
The discipline matches benchmarking a prompt-injection detector: recall, FPR, p95 latency and cost on your own benign corpus, never one headline number. A reproducible eval harness lets you rerun it when a vendor ships a new model.
Where does an AI-text detector sit in the request path?
An AI-text detector belongs beside the decision, feeding it a score, not in front of it as a blocking gate. Before: inbound text (email body, review, support ticket) passes a spam or phishing filter and lands in a queue. After: the text is normalized (NFKC, zero-width characters stripped) and scored, and the score joins sender reputation, URL analysis and account age in a risk engine. Only high-risk combinations reach human review.
The reason not to gate is the error asymmetry documented in Karr et al. (2026). At a 0.50 threshold, detectors flagged 9% to 15% of unmodified 2023-2025 research abstracts and 38% to 80% of abstracts with light AI editing, while fewer than 4% of humanizer-rewritten texts were still flagged. The authors note their labels are proxies, but the pattern is the residual risk in practice: honest users absorb the false positives, and a motivated adversary routes around the detector with an off-the-shelf rewriter.
FAQ
which ai detector is most accurate on the raid benchmark
Leidos and Pangram hold the strongest results reported by RAID’s organizers: 99.4% and 99.3% at 5% FPR on unattacked text in the COLING 2025 shared task, and 97.7% each with attacks. Vendor entries on the live leaderboard change often, so check the date and whether the entry sits in the trained-on-RAID section.
can i run the raid benchmark on my own ai detector
Yes. Install the raid-bench package from PyPI, wrap your detector as a function that returns one score per text, and run it over the unlabeled test split. Submitting predictions plus a metadata file by pull request returns an automated score at 5% FPR. The labeled train split supports local experiments at other FPR targets.
does raid test ai detectors on non-english text
Only lightly. The main train and test splits cover eight English-language domains. The extra split adds Czech news, German news and Python code, and the authors acknowledge their multilingual coverage is limited to the news domain. A RAID score says little about how a detector behaves on other languages or mixed-language content.
are ai detectors reliable enough to use as evidence
No, not as standalone evidence. RAID’s authors say detectors are not yet robust enough for high-stakes use and oppose punitive applications. A 2026 study found detectors flagged 9% to 15% of unmodified recent research abstracts while fewer than 4% of humanizer-rewritten texts were still caught. Treat a score as one signal feeding human review.
