Skip to content
Eval Suites50

LLM vulnerability scanners: OWASP coverage and blind spots

Sources checked 2 min read · By the Eval Suites editors

In brief

An LLM vulnerability scanner can probe configured model and application behaviors; it cannot establish the whole system’s security from prompt responses alone. Map the OWASP 2025 LLM Top 10 to model tests, tool permissions and infrastructure checks, then preserve evidence of each actual outcome.[1][2]

Risk taxonomy
OWASP 2025 LLM Top 10 [1]
Certification from scan
None
PII test data
Fictional, isolated records and canaries

How do I scan my LLM prompts and tools for OWASP Top 10 vulnerabilities?

Use OWASP’s versioned risk list as a coverage map, not a checklist that a model-only scan can certify. Promptfoo documents OWASP-oriented red-team testing, while several risks also require checks outside the model.[1][2]

OWASP 2025 risk names and proposed verification layers
RiskExample model/application testAdditional review
LLM01 Prompt InjectionInject untrusted fixture instructionsTool authorization and trust boundaries
LLM02 Sensitive Information DisclosureCross-session fake PII and canary checksAccess controls and log/storage policy
LLM03 Supply ChainNot resolved by a prompt scanDependency, model and artifact provenance
LLM04 Data and Model PoisoningControlled corrupted evidence casesIngestion permissions and data integrity
LLM05 Improper Output HandlingTreat model output as untrusted inputEscaping, query parameterization and execution restrictions
LLM06 Excessive AgencyAttempt an unauthorized fixture actionLeast privilege and independent confirmation
LLM07 System Prompt LeakageProbe protected fixture instruction disclosureNever store production secrets in a prompt
LLM08 Vector and Embedding WeaknessesCross-tenant retrieval and poisoned-chunk casesIndex isolation and ingestion controls
LLM09 MisinformationUnsupported answer and abstention testsReference quality and human escalation
LLM10 Unbounded ConsumptionRepeated tool calls and oversized-input testsTimeouts, quotas and spend controls

Risk names come from OWASP; the test and review columns are our proposed application checklist. They are not OWASP certification criteria or a claim of exhaustive coverage.[1]

What is the best scanner for detecting data exfiltration in tool-calling LLMs?

Choose a setup that observes tool arguments and outbound effects, not only final answer text. No reviewed evidence establishes one universal best scanner. Promptfoo can exercise configured attack cases; AgentDojo studies injection in agent environments.[3][4]

Plant a fictional canary in an authorized fixture, intercept network and email tools, and inspect every attempted destination. Assert that unauthorized calls are denied even when the model says it is complying with policy. A string-matching check cannot detect all transformed leaks, so include encoded output, cross-session retrieval and semantic review where your threat model requires them.

Keep benign tasks in the same run. A system that never retrieves evidence may stop leaks by losing the functionality you intended to protect. Record the failure case, permission state and blocked or completed effect.

Can Promptfoo scan for PII leaks and system prompt extraction?

Yes. Promptfoo documents PII plugins for direct, session, social and API/database disclosure, and a prompt-extraction plugin requiring the target systemPrompt configuration. A plugin’s verdict is evidence to inspect, not a legal compliance guarantee.[5][6]

Documented plugin IDs; use fictional target instructions and data
# Merge into a configured, authorized test-target red-team file.
redteam:
  plugins:
    - pii:direct
    - pii:session
    - pii:api-db
    - id: prompt-extraction
      config:
        systemPrompt: "Fixture assistant. Internal marker: TEST_ONLY_742."

The extraction plugin’s systemPrompt is required by the documented configuration. Don’t paste production credentials into it. For PII, distinguish unauthorized disclosure from repeating information the same user just supplied; Promptfoo’s documentation makes that distinction explicitly.[6][5]

How should scanner results change a release decision?

Base the release decision on verified forbidden effects and documented residual risk. A proposed workflow is to reproduce the case, identify the broken boundary, fix it outside the prompt when necessary and add a regression test.

  1. Validate the finding against the exact target, model and permission context.
  2. Separate a confirmed leak or unauthorized action from a judge-only allegation.
  3. Retest the fix with related benign and adversarial cases, not just the original payload.
  4. Record remaining coverage gaps, including dependencies, retrieval isolation and non-model output handling.

Related suites

Browse the free security directory

Related reading

Sources

  1. 1. 2025 LLM Top 10 · OWASP GenAI Security Project · Official docs · Sources checked
  2. 2. OWASP LLM Top 10 testing · Promptfoo · Official docs · Sources checked
  3. 3. LLM red teaming · Promptfoo · Official docs · Sources checked
  4. 4. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · arXiv:2406.13352 (ETH Zurich) · Paper · Sources checked
  5. 5. PII disclosure testing plugins · Promptfoo · Official docs · Sources checked
  6. 6. Prompt extraction plugin · Promptfoo · Official docs · Sources checked