ProofLayer's specialized attack policy met the attack objective in all 949 exported AgentDojo cases. The target agent still completed its original user task in 82.61% of attacked runs.
That combination matters. An attack that merely crashes an agent can produce a high failure count without demonstrating control. Here, the attack objective succeeded while useful agent behavior remained largely intact.
Do you know what your agents will do under attack?
ProofLayer tests your agent workflows, tools, and MCP servers. You receive verified findings with replayable attack traces.
Why AgentDojo
Agent security is not the same as chatbot safety. Agents read untrusted data, call tools, and change external state. A successful indirect prompt injection can therefore become an email, a bank transfer, a booking, or an unauthorized workspace action.
AgentDojo was created by researchers at ETH Zurich to evaluate these end-to-end failures. The original NeurIPS 2024 release introduced 97 realistic tasks and 629 security test cases. It treats the benchmark as an extensible environment for agents, attacks, and defenses—not a frozen prompt list.
Our selected v1.2.2 configuration exported 949 attack cases across Workspace, Travel, Banking, and Slack. Those domains force the target to reason over tool output while preserving the user's intended task.
How we ran the evaluation
Controlled target
AgentDojo v1.2.2 with GPT-5.2 as the target model and no defense configured.
Official scoring
Utility and security came from AgentDojo's task evaluators, not a model judging its own output.
Complete case set
949 exported attack cases plus 35 baseline logs produced 984 official logs.
Clean execution
All suites completed with zero official errors and zero malformed case logs.
Performance by suite
Workspace
560 casesTravel
140 casesBanking
144 casesSlack
105 casesOverall
949 casesAttack success was uniform: 100% in every suite. Utility was not. Workspace retained 89.46% utility, while Banking reached 67.36%. Travel and Slack landed between them at 78.57% and 72.38%.
The spread is operationally useful. It shows where an attack policy can preserve normal task completion and where the target model struggles with the underlying workflow. Aggregate scores alone hide that distinction.
Template policy vs. specialized adapter
We ran the same 949 cases with a strong template-based policy and a specialized adapter-driven policy. Both were effective. The adapter closed the four cases the template policy did not, while keeping overall utility effectively flat.
| Attack policy | Cases | Attack success | Utility | Security |
|---|---|---|---|---|
| Template policy | 949 | 99.58% | 82.40% | 0.42% |
| Adapter policy | 949 | 100.00% | 82.61% | 0.00% |
| Change | — | +0.42 pp | +0.21 pp | −0.42 pp |
The largest utility gains appeared in Travel (+7.14 percentage points) and Banking (+4.17 points). Workspace declined by 2.33 points and Slack by 0.95 points. These differences are why we report every suite instead of only the overall average.
What the results mean for security teams
A strong static library can still leave a meaningful tail. The template policy already reached 99.58% attack success. Specialization mattered in the last four cases, including the remaining Banking and Slack gaps.
Utility belongs beside attack success. Security testing should not get credit for simply disabling the agent. Measuring the original task under attack exposes whether the system remains useful while the adversarial objective succeeds.
Agent behavior must be evaluated end to end. The relevant outcome is not whether a prompt looks malicious. It is whether untrusted data changes a tool call, state transition, or external action.
Suite-level variance should guide remediation. Banking's lower utility suggests a different engineering problem than Workspace's higher utility, even though attack success was identical.
What this benchmark does not prove
It does not mean every production agent is universally vulnerable. This was one target model, one AgentDojo version, and a controlled configuration with no defense enabled.
Each policy was evaluated in one completed run. We are not reporting repeated-seed confidence intervals. AgentDojo's simulated workflows also cannot reproduce every production permission boundary, tool, memory system, or monitoring control.
The specialized adapter was built for adversarial testing. These results should be read as attack-policy performance on this benchmark—not as a general model capability score.
Benchmarks are the baseline. Your agents are the real test.
The next evaluation layer is defensive: rerun the same attack policy against prompt filters, tool authorization, data-flow controls, and runtime rules. Then repeat across models and real application traces.
Benchmarks provide a controlled comparison. Production assurance requires the same loop after every meaningful change to the model, system prompt, tool set, MCP server, permission boundary, or memory layer.
A ProofLayer red-team assessment shows which attacks succeed against your deployed system and gives your team the traces to fix them.
Sources and artifacts
- [1]E. Debenedetti et al. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS 2024.
- [2]AgentDojo documentation, including benchmark execution and pipeline concepts.
- [3]AgentDojo source repository.
- [4]ProofLayer AgentDojo v1.2.2 benchmark artifacts, completed July 24, 2026: 949 attack cases, 984 official logs, and zero execution errors.