inferwire
/
Cybersecurity·4 min read

PIMiner Red-Team Agent Reuses Prompt Injection Strategies

Penn State researchers built an agent that keeps prompt injection strategies in a library and reuses them on new models, succeeding on 76.2 percent of IPIArena tasks against Gemini 2.5 Pro.

TL;DR

  • Penn State's PIMiner stores prompt injection strategies in a library and reuses them, with up to 10 tries, against models it never trained on [^1].
  • It succeeded on 76.2 percent of IPIArena tasks against Gemini 2.5 Pro, while Claude Opus 4.5 held it to 4.8 percent [^1].

Background

In a prompt injection attack, someone hides instructions in content an AI agent reads, such as a webpage, a document or a tool's output, to change the agent's behavior [^1]. Red-teaming means attacking your own system to find such weaknesses. Automated red-teaming has followed two routes: training an attacker model with reinforcement learning, which takes many queries and transfers poorly to new targets, or searching for a working attack per task, which starts from scratch each time and succeeds less often [^1].

What happened

Yanting Wang, Chenlong Yin, Runpeng Geng and Jinyuan Jia of Pennsylvania State University posted the PIMiner paper on arXiv on August 5 [^1]. Their system turns attack history into a strategy library: Markdown files that each record where a strategy applies, an injection template, examples that worked and known failure conditions. The library starts with only a template. For each task, a router picks the three most relevant strategies, an attacker refines the injection for up to 10 rounds using notes on what failed, and a digester then updates the library, so failed attacks also refine it [^1].

The authors trained the library on eight combinations of two benchmarks, AgentDojo and IPIArena, with GPT-5-nano, GPT-5, Claude Haiku 4.5 and Claude Sonnet 4.5 as targets, using Claude Opus 4.7 to run PIMiner [^1]. Against nine models, it succeeded within 10 tries on IPIArena in 76.2 percent of cases against Gemini 2.5 Pro, 61.9 percent against GPT-5.1 and 42.9 percent against Claude Sonnet 4.5. On AgentDojo the rates were 86.7, 53.3 and 40.0 percent [^1]. DeepSeek-V4-Pro was the most vulnerable target on AgentDojo at 93.3 percent, and Claude Opus 4.5 the most robust, at 4.8 percent on IPIArena and 3.3 percent on AgentDojo [^1]. On its first try against Sonnet 4.5, PIMiner succeeded 28.6 percent of the time; human attackers in IPIArena's own study averaged 1.0 percent [^1].

Against reinforcement learning methods, PIMiner matched RL-Hammer and PISmith on the InjecAgent benchmark with a success rate of 1.0 on all three targets [^1]. On AgentDojo it beat PISmith on GPT-5-nano (0.53 against 0.38), came close on GPT-4o-mini (0.73 against 0.78) and fell behind on GPT-4.1-nano (0.63 against 0.81). PISmith, however, trains a separate attacker for each target model [^1]. With a Claude Code subscription covering the agents, the authors put the extra API cost of training at about $20, while reinforcement learning methods often need around 10,000 target queries, which can cost more than $100 for a model like GPT-5 [^1].

Why it matters

The authors see two uses: testing how exposed an agent is before deployment, and collecting successful attacks as training data for defenses such as guardrail models [^1]. A cheap attacker that carries over to new models makes that testing easier to repeat each time a model changes. The library is also readable. Two strategies make up about 80 percent of its examples, and one of them, Forged Chat Turn, uses role-delimiter tokens to fake a new user turn, so the target treats the attacker's text as the user's instruction [^1].

The results have limits. The test sets are small, with 21 IPIArena and 30 AgentDojo samples [^1]. The study measures how well the models themselves resist injection, not external defenses such as detectors or input filters [^1]. The authors acknowledge that such a tool is dual-use; they note that its strategies build on attack techniques already described in public research and release the code as a research evaluation tool [^1].

Practical example

Say you run a support agent that reads customer emails and can call a refund tool, and you're about to switch the model behind it. Before the switch, you want to know how the new model handles instructions hidden in an email. A tool like PIMiner, whose code the authors published, could run its stored strategies against a test copy of your agent with up to 10 tries per case [^1]. The spread in the paper, from 4.8 to 81.0 percent success on the same IPIArena benchmark depending on the model, shows why that test is worth running before the switch, not after [^1].

Related gear

We recommend this book because it teaches a structured way to find and rank the threats against a system before attackers do.

AdvertisementAmazon

Threat Modeling

Sources

  1. [1]arXiv — Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming