Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation

Reported by arXiv cs.CR (Chetan Pathade, Prathamesh Pawar, Shubham Patil)

Summary by Cyber Security News23 Sep 2026, 06:00 CEST

About this summary. This is a short, independently written summary of an article first published by arXiv cs.CR. Cyber Security News did not report or verify the underlying story. Read the original: https://arxiv.org/abs/2609.25173

arXiv:2609.25173v1 Announce Type: new Abstract: Attack success rate (ASR) is the headline metric in nearly every published evaluation of attacks on, and defenses for, LLM agents. We argue that ASR as currently used is not a single quantity but a family of metrics parameterized by six design choices that papers seldom specify and never hold constant across the literature. We support this with two studies that require no proprietary access.

Read the full article at arXiv cs.CR →

Source attribution: headline and facts are from arXiv cs.CR (arxiv.org). Summary method: excerpt of the source description. See our source attribution policy.

Related coverage

AI-powered fuzzing with the GitHub Security Lab Taskflow Agent

New Browser Guard features add protection before and after you click

Fwd: Tor Project Forum: Security Release 0.4.9.13

ACTS: A multi-tier benchmark evaluating LLM cipher identification under controlled blind conditions

Solidity Meets LLMs: A Transformer-Based Approach to Smart Contract Vulnerability Detection

The Like Trap: Multi-Stage Poisoning against Agents in Similarity-based Recommendation Systems