CyberBinGym finds the best AI agent locates the flaw behind a binary security patch in 88% of tasks and exploits 20%
arXiv · Research preprint
Read the source (opens in a new tab)- What happened
- Researchers from the National University of Singapore, UC Berkeley, UC Riverside and UC Santa Barbara posted CyberBinGym on November 7, 2026, a benchmark of whether AI agents can turn compiled patches into exploits.
- Key points
- Agents got stripped vulnerable and patched binaries, with no source code, for 100 tasks drawn from real vulnerabilities in 53 widely used projects.
- OpenAI Codex with GPT-5.6 recovered the vulnerability in 88% of tasks and exploited 20%; Claude Code with Claude Opus 5 managed 85% and 6%, Kimi K3 64% and none.
- With standard binary hardening enabled, the strongest agent still exploited two of its 20 previously successful tasks.
- The authors say these capabilities "compress defenders' effective response window, increasing the urgency of timely patch deployment."
- Evidence
- Research preprint, not peer reviewed: a controlled benchmark with trajectory audits to reject reward hacking. The authors say exploitation "remains unreliable." Main runs used trusted-access programs with model safeguards disabled.