A study published on arXiv in April 2026, “Evaluation of Prompt Injection Defenses in Large Language Models,” ran an adaptive attacker against nine different defense configurations across more than 20,000 attacks. Its conclusion: every defense that relies on the model itself to resist injected instructions eventually broke. Only one approach held — filtering the model’s output through hardcoded rules in separate application code, which the paper reports achieved zero leaks across 15,000 attacks.
How the test worked
The researchers, from Swept AI and the University of Michigan, tested three models — Gemini 2.5 Pro, GPT-5.4, and Claude Sonnet 4.6 — against nine defense configurations, ranging from a no-defense baseline up to a combined multi-layer setup:
- No defense (baseline)
- Security directives (prompting the model to resist injection)
- Input sanitization
- Delimiter-based isolation of untrusted content
- Instruction hierarchy
- Output filtering (checked outside the model)
- Sandwich defense (instructions repeated before and after untrusted content)
- Multi-layer (all defenses combined)
- Multi-layer without output filtering (an ablation to isolate its contribution)
An adaptive attacker evolved its approach across hundreds of rounds against each configuration, rather than running a fixed, static set of attack prompts.
What the models had in common
With no defense and default safety settings, the attacker reached a peak severity score of 1.00 against both Gemini 2.5 Pro and GPT-5.4 by round 10. Claude Sonnet 4.6 held out longer, reaching a peak score of 0.90 by round 300 — better, but still compromised. Extended to 500 rounds, every defense that relied on the model policing itself reached critical severity (a score of 0.9 or higher) within 300 rounds, regardless of which model it ran on. Only output filtering and the full multi-layer configuration (which includes output filtering) held at a 0% leak rate across all 500 rounds.
The paper’s recommendation
The authors’ conclusion is blunt: security boundaries have to be enforced in application code, not by the model being attacked. They go further, recommending that AI systems handling sensitive operations be restricted to internal, trusted personnel until defenses undergo formal verification.
What’s confirmed and what isn’t
- Confirmed: the attack scale, the nine defense configurations, the three models tested, and the reported severity scores and leak rates, per the paper’s own text on arXiv.
- Confirmed: the authors’ affiliation with Swept AI and the University of Michigan.
- Not independently verified: The Signal has not reproduced these results or seen a third-party replication; the paper is a preprint and, as of publication, has not been confirmed as peer-reviewed.
