Automated LLM red teaming: tools and a bounded CI workflow
Sources checked 2 min read · By the Eval Suites editors
In brief
Automated LLM red teaming generates or runs adversarial cases against an authorized test system. Promptfoo offers configured red-team scans, PyRIT is a programmable risk-testing framework, and Inspect AI provides evaluation tasks and agent execution infrastructure. None replaces application-specific security review.[1][2][3]
- Authorization
- Only systems you own or have permission to test
- Target environment
- Isolated fixture data and restricted tools
- Release evidence
- Attack outcome plus normal-task utility
What open-source tools exist for automated LLM red teaming?
Start with Promptfoo for configuration-led scans, PyRIT for programmable campaigns, or Inspect AI for custom scored agent tasks. Choose the execution model you can maintain, rather than counting plugins.[1][2][3]
| Tool | Practical fit | Boundary |
|---|---|---|
| Promptfoo | Configured adversarial generation and evaluation | Must connect to your application and relevant attack surfaces |
| PyRIT | Python risk-identification workflows | Campaign design and scoring remain your responsibility |
| Inspect AI | Tasks, scorers and agent evaluation infrastructure | Not a turnkey vulnerability scanner |
Inspect’s sandboxing can isolate tool execution when a task needs a real environment. Restrict network access and credentials too; a container alone is not permission to contact production systems.[4]
How do I automate prompt injection testing in a CI/CD pipeline?
Run a small frozen regression set on pull requests and a separately budgeted generation campaign on a schedule. Promptfoo documents CI integration and red-team execution; freezing reviewed cases is our proposed control for reproducible merge decisions.[5][1]
- Deploy a test endpoint backed by fixtures, not customer records. Disable real email, payment and external write tools.
- Generate attacks in a controlled campaign, review their goals and expected checks, and save approved cases in version control.
- On a pull request, run the same approved cases against the candidate and baseline. Record model, prompt, dataset and scorer versions.
- Set job timeouts, target request quotas, concurrency limits and a spend cap before a scheduled campaign starts. Count generator and judge calls as well as target calls.
- Block on confirmed forbidden side effects. Route ambiguous judge verdicts to review, and retain a benign suite so a refusal-only defense cannot look successful.
# In an authorized test environment, after installing and pinning Promptfoo:
promptfoo redteam init
promptfoo redteam run
# Review generated cases and results before promoting them to a PR gate.These commands don’t define your endpoint, model budget or release threshold. Configure those for your system first. Protect CI secrets from untrusted pull requests and redact logs before storing artifacts.[1]
What is the difference between direct and indirect prompt injection attacks?
Direct injection arrives through the user’s prompt; indirect injection arrives through content the application treats as data, such as a retrieved document or tool response. OWASP separates these routes because the attacker may never talk to the agent directly.[6]
| Entry point | Controlled test | Expected boundary |
|---|---|---|
| User prompt | Request an action outside the user’s permissions | Authorization still denies the tool call |
| Retrieved document | Place a conflicting instruction inside fixture text | Document is treated as evidence, not authority |
| Tool response | Add a request to forward a fixture secret | No secret leaves output or allowed tools |
AgentDojo evaluates prompt injection in agent environments, making it useful for studying attacks delivered through tool results rather than only chat messages.[7]
What should an automated red-team report prove?
A useful report shows which attacker goal occurred and whether the normal task still worked. AgentDojo’s attacks-and-defenses evaluation motivates checking both outcomes.[7]
For a fixture email agent, assert that the intended draft exists and that no unauthorized destination received a message. Inspect the tool trace and final state, not just the assistant’s refusal. Repeat suspicious cases, preserve exact inputs and separate a verified failure from a model judge’s suggestion. A scan passing today doesn’t certify future models or changed tools.
Related suites
Related reading
Sources
- 1. LLM red teaming · Promptfoo · Official docs · Sources checked
- 2. Python Risk Identification Tool · Microsoft · Repository · Sources checked
- 3. Inspect AI evaluation framework · UK AI Security Institute · Official docs · Sources checked
- 4. Inspect AI: Sandboxing · UK AI Security Institute · Official docs · Sources checked
- 5. GitHub Action integration · Promptfoo · Official docs · Sources checked
- 6. LLM01: Prompt injection · OWASP GenAI Security Project · Official docs · Sources checked
- 7. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · arXiv:2406.13352 (ETH Zurich) · Paper · Sources checked