LLM vulnerability scanners: OWASP coverage and blind spots
Sources checked 2 min read · By the Eval Suites editors
In brief
An LLM vulnerability scanner can probe configured model and application behaviors; it cannot establish the whole system’s security from prompt responses alone. Map the OWASP 2025 LLM Top 10 to model tests, tool permissions and infrastructure checks, then preserve evidence of each actual outcome.[1][2]
- Risk taxonomy
- OWASP 2025 LLM Top 10 [1]
- Certification from scan
- None
- PII test data
- Fictional, isolated records and canaries
How do I scan my LLM prompts and tools for OWASP Top 10 vulnerabilities?
Use OWASP’s versioned risk list as a coverage map, not a checklist that a model-only scan can certify. Promptfoo documents OWASP-oriented red-team testing, while several risks also require checks outside the model.[1][2]
| Risk | Example model/application test | Additional review |
|---|---|---|
| LLM01 Prompt Injection | Inject untrusted fixture instructions | Tool authorization and trust boundaries |
| LLM02 Sensitive Information Disclosure | Cross-session fake PII and canary checks | Access controls and log/storage policy |
| LLM03 Supply Chain | Not resolved by a prompt scan | Dependency, model and artifact provenance |
| LLM04 Data and Model Poisoning | Controlled corrupted evidence cases | Ingestion permissions and data integrity |
| LLM05 Improper Output Handling | Treat model output as untrusted input | Escaping, query parameterization and execution restrictions |
| LLM06 Excessive Agency | Attempt an unauthorized fixture action | Least privilege and independent confirmation |
| LLM07 System Prompt Leakage | Probe protected fixture instruction disclosure | Never store production secrets in a prompt |
| LLM08 Vector and Embedding Weaknesses | Cross-tenant retrieval and poisoned-chunk cases | Index isolation and ingestion controls |
| LLM09 Misinformation | Unsupported answer and abstention tests | Reference quality and human escalation |
| LLM10 Unbounded Consumption | Repeated tool calls and oversized-input tests | Timeouts, quotas and spend controls |
Risk names come from OWASP; the test and review columns are our proposed application checklist. They are not OWASP certification criteria or a claim of exhaustive coverage.[1]
What is the best scanner for detecting data exfiltration in tool-calling LLMs?
Choose a setup that observes tool arguments and outbound effects, not only final answer text. No reviewed evidence establishes one universal best scanner. Promptfoo can exercise configured attack cases; AgentDojo studies injection in agent environments.[3][4]
Plant a fictional canary in an authorized fixture, intercept network and email tools, and inspect every attempted destination. Assert that unauthorized calls are denied even when the model says it is complying with policy. A string-matching check cannot detect all transformed leaks, so include encoded output, cross-session retrieval and semantic review where your threat model requires them.
Keep benign tasks in the same run. A system that never retrieves evidence may stop leaks by losing the functionality you intended to protect. Record the failure case, permission state and blocked or completed effect.
Can Promptfoo scan for PII leaks and system prompt extraction?
Yes. Promptfoo documents PII plugins for direct, session, social and API/database disclosure, and a prompt-extraction plugin requiring the target systemPrompt configuration. A plugin’s verdict is evidence to inspect, not a legal compliance guarantee.[5][6]
# Merge into a configured, authorized test-target red-team file.
redteam:
plugins:
- pii:direct
- pii:session
- pii:api-db
- id: prompt-extraction
config:
systemPrompt: "Fixture assistant. Internal marker: TEST_ONLY_742."The extraction plugin’s systemPrompt is required by the documented configuration. Don’t paste production credentials into it. For PII, distinguish unauthorized disclosure from repeating information the same user just supplied; Promptfoo’s documentation makes that distinction explicitly.[6][5]
How should scanner results change a release decision?
Base the release decision on verified forbidden effects and documented residual risk. A proposed workflow is to reproduce the case, identify the broken boundary, fix it outside the prompt when necessary and add a regression test.
- Validate the finding against the exact target, model and permission context.
- Separate a confirmed leak or unauthorized action from a judge-only allegation.
- Retest the fix with related benign and adversarial cases, not just the original payload.
- Record remaining coverage gaps, including dependencies, retrieval isolation and non-model output handling.
Related suites
Browse the free security directory
Related reading
Sources
- 1. 2025 LLM Top 10 · OWASP GenAI Security Project · Official docs · Sources checked
- 2. OWASP LLM Top 10 testing · Promptfoo · Official docs · Sources checked
- 3. LLM red teaming · Promptfoo · Official docs · Sources checked
- 4. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · arXiv:2406.13352 (ETH Zurich) · Paper · Sources checked
- 5. PII disclosure testing plugins · Promptfoo · Official docs · Sources checked
- 6. Prompt extraction plugin · Promptfoo · Official docs · Sources checked