Skip to content
Eval Suites50

How to Evaluate LLM Security Against Prompt Injection & Dual-Control Attacks

Published Sources checked 3 min read · By the Eval Suites editors

At-a-glance answer (TL;DR)

Test prompt injection where it actually enters your system: tool outputs, retrieved documents, emails, and web pages, not only the chat box. Score two numbers together, task success without attack and attack success rate, because a defense that blocks attacks by breaking the product isn't a fix. For agents that share control with a user, add cases where the user side is the one being manipulated.

What prompt injection is, in testing terms

Prompt injection is input that changes what the model does in ways the developer didn't intend. OWASP lists it as LLM01 and separates direct injection, typed by the user, from indirect injection, which arrives inside content the model reads, like a web page or file.[1]

Indirect injection is the one that hurts agents. Greshake and colleagues showed attackers can plant instructions in data an LLM-integrated app will retrieve, then trigger data theft or unwanted actions without ever talking to the model directly.[2]

Where to inject test payloads

  • Tool and API responses, especially free-text fields like notes, descriptions, and error messages.
  • Retrieved documents in your RAG index.
  • Inbound email, tickets, calendar invites, and chat messages the agent summarizes.
  • Web pages and PDFs a browsing agent opens.
  • File names, image alt text, and metadata that get pasted into prompts.

AgentDojo is built around this idea: it pairs 97 realistic agent tasks with 629 security test cases where injections arrive through tool results, so you can see whether the agent still finishes the user's job while ignoring the planted instruction.[3]

How to score a prompt injection eval

Track utility and attack success side by side. A system that refuses everything has a 0% attack success rate and no users.

Prompt injection eval scorecard
MeasureQuestion it answersHow to check it
Benign utilityDoes the agent finish normal tasks?End-state check on tasks with no injection
Utility under attackDoes it still finish the user's task when an injection is present?Same end-state check, injected environment
Attack success rateDid the attacker's goal happen?Check for the attacker's side effect: email sent, file changed, data in a URL
Leak checksDid secrets or PII appear in output or tool arguments?Regex and exact-match on canaries planted in context

Score the attacker's goal by checking side effects, not the model's words. An agent can say "I won't do that" and call the tool anyway.

What "dual-control" means for security testing

Dual-control describes environments where both the agent and a user can act on shared state. τ²-bench introduced it to evaluate agents that have to guide a user through actions on the user's own device.[4] It's a reliability benchmark, not a security one, but the setup exposes a real attack surface: the agent trusts the user's reports about what they did.

  • The simulated user lies about completing a step to push the agent past a verification gate.
  • The user pastes content that contains instructions aimed at the agent.
  • The user asks the agent to perform an action on their behalf that policy reserves for the account holder.

These are test ideas we derived from the dual-control design, not results reported by its authors.

Tools that generate and run injection tests

Promptfoo's red-team mode generates adversarial inputs, including prompt injection and jailbreak strategies, against your app and reports what got through.[5] For agents with tools, run AgentDojo-style environments in addition, since generated chat attacks don't reach the tool-output surface.

Start a Promptfoo red-team scan
npx promptfoo@latest redteam init
npx promptfoo@latest redteam run

Frequently asked questions

Suites mentioned in this guide

Related guides

Sources

  1. 1. LLM01: Prompt Injection · OWASP GenAI Security Project · Official docs · Sources checked
  2. 2. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection · arXiv:2302.12173 · Paper · Sources checked
  3. 3. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · arXiv:2406.13352 (ETH Zurich) · Paper · Sources checked
  4. 4. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment · arXiv:2506.07982 (Sierra) · Paper · Sources checked
  5. 5. Promptfoo: LLM red teaming · Promptfoo · Official docs · Sources checked