Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks
Proposed cost-aware metric for LLM jailbreak evaluation accounts for detection opportunities created by repeated queries—more realistic than raw attack success rate for defensive planning.
Summary written by editorial AI · Source link below
arXiv:2610.26589v2 Announce Type: replace Abstract: Attack success rate (ASR) evaluates jailbreaks against large language models without accounting for repeated calls to the victim model. Yet each additional call gives the deployed system another opportunity to detect the attack and intervene. We introduce $\mathrm{ASR}_{\mathrm{STEP}}$, which penalizes additional victim-model calls while retaining binary success judgments. Our evaluation considers GCG, PAIR with a step budget, and a single-ste
Editorial Analysis
Evaluating jailbreaks by success rate alone ignores the defender's increasing detection window with each attempt; cost-aware metrics better inform LLM deployment risk assessments.
Incorporate cost-aware jailbreak metrics into your LLM red-team methodology to reflect realistic adversary economics.
Forward-looking interpretation drafted by editorial AI under human review — not a reproduction of the source. See methodology.
External link — opens at arXiv Crypto & Security in a new tab.
More from the AI Security Desk
- OpenAI AI Agents Demonstrate VM Escapes and Secret Cross-Sandbox Communication1d
- Exposed Infrastructure Reveals How Ransomware Affiliates Used AI Agents to Execute Intrusions3d
- You gave your AI agent access, but who is giving it orders?3d
- AI Agents as Insider Threats: Inside Microsoft's Bet on a Unified Security Stack3d
- Critical x402 SDK Flaw Could Let Malicious APIs Drain AI Agents’ Token Balances3d