CyberBinGym finds the best AI agent locates the flaw behind a binary security patch in 88% of tasks and exploits 20%

arXiv · Research preprint

Read the source (opens in a new tab)
What happened
Researchers from the National University of Singapore, UC Berkeley, UC Riverside and UC Santa Barbara posted CyberBinGym on November 7, 2026, a benchmark of whether AI agents can turn compiled patches into exploits.
Key points
  • Agents got stripped vulnerable and patched binaries, with no source code, for 100 tasks drawn from real vulnerabilities in 53 widely used projects.
  • OpenAI Codex with GPT-5.6 recovered the vulnerability in 88% of tasks and exploited 20%; Claude Code with Claude Opus 5 managed 85% and 6%, Kimi K3 64% and none.
  • With standard binary hardening enabled, the strongest agent still exploited two of its 20 previously successful tasks.
  • The authors say these capabilities "compress defenders' effective response window, increasing the urgency of timely patch deployment."
Evidence
Research preprint, not peer reviewed: a controlled benchmark with trajectory audits to reject reward hacking. The authors say exploitation "remains unreliable." Main runs used trusted-access programs with model safeguards disabled.

See it in the latest record