How to Evaluate LLM Security Against Prompt Injection & Dual-Control Attacks
Published Sources checked 3 min read · By the Eval Suites editors
What prompt injection is, in testing terms
Prompt injection is input that changes what the model does in ways the developer didn't intend. OWASP lists it as LLM01 and separates direct injection, typed by the user, from indirect injection, which arrives inside content the model reads, like a web page or file.[1]
Indirect injection is the one that hurts agents. Greshake and colleagues showed attackers can plant instructions in data an LLM-integrated app will retrieve, then trigger data theft or unwanted actions without ever talking to the model directly.[2]
Where to inject test payloads
- Tool and API responses, especially free-text fields like notes, descriptions, and error messages.
- Retrieved documents in your RAG index.
- Inbound email, tickets, calendar invites, and chat messages the agent summarizes.
- Web pages and PDFs a browsing agent opens.
- File names, image alt text, and metadata that get pasted into prompts.
AgentDojo is built around this idea: it pairs 97 realistic agent tasks with 629 security test cases where injections arrive through tool results, so you can see whether the agent still finishes the user's job while ignoring the planted instruction.[3]
How to score a prompt injection eval
Track utility and attack success side by side. A system that refuses everything has a 0% attack success rate and no users.
| Measure | Question it answers | How to check it |
|---|---|---|
| Benign utility | Does the agent finish normal tasks? | End-state check on tasks with no injection |
| Utility under attack | Does it still finish the user's task when an injection is present? | Same end-state check, injected environment |
| Attack success rate | Did the attacker's goal happen? | Check for the attacker's side effect: email sent, file changed, data in a URL |
| Leak checks | Did secrets or PII appear in output or tool arguments? | Regex and exact-match on canaries planted in context |
Score the attacker's goal by checking side effects, not the model's words. An agent can say "I won't do that" and call the tool anyway.
What "dual-control" means for security testing
Dual-control describes environments where both the agent and a user can act on shared state. τ²-bench introduced it to evaluate agents that have to guide a user through actions on the user's own device.[4] It's a reliability benchmark, not a security one, but the setup exposes a real attack surface: the agent trusts the user's reports about what they did.
- The simulated user lies about completing a step to push the agent past a verification gate.
- The user pastes content that contains instructions aimed at the agent.
- The user asks the agent to perform an action on their behalf that policy reserves for the account holder.
These are test ideas we derived from the dual-control design, not results reported by its authors.
Tools that generate and run injection tests
Promptfoo's red-team mode generates adversarial inputs, including prompt injection and jailbreak strategies, against your app and reports what got through.[5] For agents with tools, run AgentDojo-style environments in addition, since generated chat attacks don't reach the tool-output surface.
npx promptfoo@latest redteam init
npx promptfoo@latest redteam runFrequently asked questions
Suites mentioned in this guide
Related guides
Sources
- 1. LLM01: Prompt Injection · OWASP GenAI Security Project · Official docs · Sources checked
- 2. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection · arXiv:2302.12173 · Paper · Sources checked
- 3. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · arXiv:2406.13352 (ETH Zurich) · Paper · Sources checked
- 4. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment · arXiv:2506.07982 (Sierra) · Paper · Sources checked
- 5. Promptfoo: LLM red teaming · Promptfoo · Official docs · Sources checked