Skip to content
Eval Suites50

Automated LLM red teaming: tools and a bounded CI workflow

Sources checked 2 min read · By the Eval Suites editors

In brief

Automated LLM red teaming generates or runs adversarial cases against an authorized test system. Promptfoo offers configured red-team scans, PyRIT is a programmable risk-testing framework, and Inspect AI provides evaluation tasks and agent execution infrastructure. None replaces application-specific security review.[1][2][3]

Authorization
Only systems you own or have permission to test
Target environment
Isolated fixture data and restricted tools
Release evidence
Attack outcome plus normal-task utility

What open-source tools exist for automated LLM red teaming?

Start with Promptfoo for configuration-led scans, PyRIT for programmable campaigns, or Inspect AI for custom scored agent tasks. Choose the execution model you can maintain, rather than counting plugins.[1][2][3]

Documented roles, not a measured speed ranking
ToolPractical fitBoundary
PromptfooConfigured adversarial generation and evaluationMust connect to your application and relevant attack surfaces
PyRITPython risk-identification workflowsCampaign design and scoring remain your responsibility
Inspect AITasks, scorers and agent evaluation infrastructureNot a turnkey vulnerability scanner

Inspect’s sandboxing can isolate tool execution when a task needs a real environment. Restrict network access and credentials too; a container alone is not permission to contact production systems.[4]

How do I automate prompt injection testing in a CI/CD pipeline?

Run a small frozen regression set on pull requests and a separately budgeted generation campaign on a schedule. Promptfoo documents CI integration and red-team execution; freezing reviewed cases is our proposed control for reproducible merge decisions.[5][1]

  1. Deploy a test endpoint backed by fixtures, not customer records. Disable real email, payment and external write tools.
  2. Generate attacks in a controlled campaign, review their goals and expected checks, and save approved cases in version control.
  3. On a pull request, run the same approved cases against the candidate and baseline. Record model, prompt, dataset and scorer versions.
  4. Set job timeouts, target request quotas, concurrency limits and a spend cap before a scheduled campaign starts. Count generator and judge calls as well as target calls.
  5. Block on confirmed forbidden side effects. Route ambiguous judge verdicts to review, and retain a benign suite so a refusal-only defense cannot look successful.
Documented red-team entry points; target configuration is required
# In an authorized test environment, after installing and pinning Promptfoo:
promptfoo redteam init
promptfoo redteam run
# Review generated cases and results before promoting them to a PR gate.

These commands don’t define your endpoint, model budget or release threshold. Configure those for your system first. Protect CI secrets from untrusted pull requests and redact logs before storing artifacts.[1]

What is the difference between direct and indirect prompt injection attacks?

Direct injection arrives through the user’s prompt; indirect injection arrives through content the application treats as data, such as a retrieved document or tool response. OWASP separates these routes because the attacker may never talk to the agent directly.[6]

Proposed fixture tests for each entry point
Entry pointControlled testExpected boundary
User promptRequest an action outside the user’s permissionsAuthorization still denies the tool call
Retrieved documentPlace a conflicting instruction inside fixture textDocument is treated as evidence, not authority
Tool responseAdd a request to forward a fixture secretNo secret leaves output or allowed tools

AgentDojo evaluates prompt injection in agent environments, making it useful for studying attacks delivered through tool results rather than only chat messages.[7]

What should an automated red-team report prove?

A useful report shows which attacker goal occurred and whether the normal task still worked. AgentDojo’s attacks-and-defenses evaluation motivates checking both outcomes.[7]

For a fixture email agent, assert that the intended draft exists and that no unauthorized destination received a message. Inspect the tool trace and final state, not just the assistant’s refusal. Repeat suspicious cases, preserve exact inputs and separate a verified failure from a model judge’s suggestion. A scan passing today doesn’t certify future models or changed tools.

Related suites

Related reading

Sources

  1. 1. LLM red teaming · Promptfoo · Official docs · Sources checked
  2. 2. Python Risk Identification Tool · Microsoft · Repository · Sources checked
  3. 3. Inspect AI evaluation framework · UK AI Security Institute · Official docs · Sources checked
  4. 4. Inspect AI: Sandboxing · UK AI Security Institute · Official docs · Sources checked
  5. 5. GitHub Action integration · Promptfoo · Official docs · Sources checked
  6. 6. LLM01: Prompt injection · OWASP GenAI Security Project · Official docs · Sources checked
  7. 7. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · arXiv:2406.13352 (ETH Zurich) · Paper · Sources checked