How Much Does the Harness Matter, and for Whom? A Multi-Model Study of Context in LLM-Driven Vulnerability Discovery
Multi-model study finds that the orchestration harness matters as much as the underlying LLM for vulnerability discovery, cautioning enterprises against model-centric tool evaluations.
Summary written by editorial AI · Source link below
arXiv:2610.28802v1 Announce Type: new Abstract: How much does an agent harness affect vulnerability discovery, and does its value depend on the model? We study 24 model-runner configurations using an open-source harness on a production codebase whose vulnerabilities were non-public at scan time. The pipeline combines static screening, LLM investigation, model self-revalidation, root-cause mapping, independent source review, and isolated proof-of-concept confirmation. Across 265 confirmed secu
Editorial Analysis
Enterprises investing in AI-assisted vulnerability scanning should evaluate the full tool architecture — not just the model — to avoid overestimating detection capability.
Require harness-level benchmarking alongside model benchmarks when procuring or building LLM-based vulnerability discovery tools.
Forward-looking interpretation drafted by editorial AI under human review — not a reproduction of the source. See methodology.
External link — opens at arXiv Crypto & Security in a new tab.
More from the Research Desk
- Signed, Sealed, and Relayed, NTLM-style: Bypassing LDAP Signing Through Reauthentication3d
- Not Where You Think: Protecting Location Privacy Against Reasoning-Based Geolocation4d
- Social-HUMINT: An LLM-Augmented Unified Framework for Measuring Social Media Exposure4d
- Practical Auditing of Differentially Private Graph Release Mechanisms: Methodology, Toolkit, and Case Study4d
- SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence4d