Open Source LLM Security Scanners: A Practitioner's Field Guide

The market for open source llm security scanners is maturing fast, but the category is still fragmented: some tools probe for vulnerabilities at model evaluation time, others enforce guardrails at runtime inference, and a few try to cover both. Picking the wrong one, or treating any single tool as sufficient, is the primary failure mode practitioners encounter. This guide maps the four most active open source projects by capability, architectural layer, and maintenance status as of mid-2026 — with a candid note on which tools have recently gone dark.

Capability Map

Garak (github.com/NVIDIA/garak, Apache 2.0, approximately 8,700 GitHub stars) is the closest thing to a purpose-built vulnerability scanner in this space. It runs static, dynamic, and adaptive probes across 100+ attack modules targeting OWASP LLM01 (Prompt Injection), hallucination, data leakage, toxic output generation, malware code production, and jailbreaks including DAN-style attacks. It supports HuggingFace, OpenAI, AWS Bedrock, Replicate, Cohere, and Groq endpoints natively, meaning it can scan third-party APIs the same way it scans a locally-hosted model. The output is a structured report mapping each probe to a pass/fail result. Think of garak as the Nmap of LLM security: it tells you what the attack surface looks like; it does not harden it.

NeMo Guardrails (github.com/NVIDIA/NeMo-Guardrails, Apache 2.0, approximately 6,800 stars, current release v0.23.0) is a runtime control, not a scanner. It sits inline on the inference path and enforces five rail types: input, dialog, retrieval, execution, and output. Built-in capabilities include jailbreak detection, content safety classification, hallucination detection, fact-checking, and moderation. The retrieval rail is architecturally notable: it can intercept and inspect retrieved context chunks before they are appended to the prompt, which addresses the retrieval-poisoning surface relevant to OWASP LLM03 and LLM05. It integrates with LangChain and supports GPT-3.5/4, LLaMA 2, Falcon, and compatible REST endpoints.

PyRIT (github.com/microsoft/PyRIT, MIT, approximately 4,200 stars) is Microsoft’s Python-based red-teaming framework. Where garak focuses on automated probe sweeps, PyRIT is an orchestration layer: you define adversarial scenarios, it automates multi-turn attack sequences, and it collects results for analyst review. The framework migrated from an archived Azure-org repository in March 2026 and now lives under the Microsoft organization with active development continuing there. It is better suited to targeted red-team engagements than continuous CI/CD scanning.

ARTKIT (github.com/BCG-X-Official/artkit) from BCG X extends adversarial testing into multi-turn human-in-the-loop simulation. It is the right tool when you need to stress-test agentic workflows with complex jailbreak sequences that require back-and-forth context rather than single-shot probes. The community is smaller than garak or NeMo Guardrails, and the project is still maturing.

A word on LLM Guard: Protect AI’s LLM Guard was a widely-referenced runtime scanner with 15 input scanners (including PromptInjection, Secrets, and PII anonymization) and 20 output scanners. It was archived on July 9, 2026 and is no longer receiving updates. Existing deployments should plan migration to an actively maintained alternative before depending on it in a production pipeline.

Where These Tools Sit in the Request Path

The inference pipeline for a production LLM application has three natural security checkpoints: before the prompt reaches the model (input), during context assembly for RAG (retrieval), and before the response is returned to the user (output). Garak and PyRIT operate outside the request path — they are assessment tools run against the model or application in a test or staging environment. NeMo Guardrails operates inside the request path as inline middleware, adding latency proportional to which rails are enabled. ARTKIT also operates outside production, during structured red-team exercises.

The practical implication is that these tools do not substitute for each other. A defensible LLM security posture requires both pre-deployment scanning (garak, PyRIT) and runtime enforcement (NeMo Guardrails or a comparable control). For an overview of runtime guardrail options beyond open source, guardml.io covers commercial and hybrid alternatives. For a detailed treatment of how prompt injection attacks exploit the gap between input validation and model behavior, aisec.blog maintains a running catalog of techniques and disclosures.

Trade-offs

Garak delivers high recall on known probe categories and is the most CI/CD-friendly option. The main cost is configuration investment: setting up scan pipelines, selecting relevant probes for your model type, and suppressing false positives on benign inputs. Custom probe authoring has a steep learning curve, but the default probe set is operationally useful without customization.

NeMo Guardrails adds inference latency in direct proportion to which rails you activate and whether any rail invokes an external classifier call. Enabling all five rail types on every request will produce meaningful p95 latency overhead in latency-sensitive applications. The Colang domain-specific language used to define dialog and topic rails is expressive but requires time to learn. Best practice is to benchmark it against your specific model and traffic shape before committing to a deployment architecture.

PyRIT is not designed for high-volume continuous scanning. It shines in scenario-driven red-team work where you need to document multi-turn attack chains for compliance reporting or internal security reviews. Throughput is low relative to garak.

ARTKIT is the most flexible for complex agentic attack simulations but comes with the highest setup burden and the smallest community for troubleshooting support.

Across all four tools, no single project covers the full OWASP LLM Top 10 surface — particularly LLM08 (Excessive Agency), LLM09 (Overreliance), and LLM10 (Model Theft), which require controls at the architecture and access-control layer, not the scanner layer.

Who Should Use What

Pick garak if you need integration-ready LLM vulnerability scanning that runs in CI/CD, you manage multiple model endpoints across providers, or you want structured OWASP LLM coverage with minimal configuration and a large active community behind it.

Pick NeMo Guardrails if you are building or operating a production LLM application and need inline runtime enforcement — particularly if RAG is in your stack and retrieval-context isolation matters.

Pick PyRIT if you are running formal red-team engagements against a generative AI application and need an automation harness that produces structured evidence for compliance documentation or penetration testing reports.

Avoid LLM Guard for new deployments — it is archived and will not receive security updates. Any existing deployment depends on a frozen codebase.

Layer, do not choose. The teams getting the most coverage are running garak or PyRIT in staging, NeMo Guardrails in production, and reviewing findings against the OWASP LLM Top 10 as a completeness checklist. No single open source scanner closes all ten risk categories.

Scanners find problems; runtime filters stop them, and the two are procured separately. For the commercial side of the request path, see the AI guardrail tools review and the wider AI security tools directory. For the assessment these scanners are one step inside, see AI security testing: a method for LLM and agent systems, and use the Scanner Tradeoff Explorer to filter published operating points against your own latency, cost, and false-positive budget.