Irregular reports Gemini 4 Argon solved all 12 multi-stage offensive cyber scenarios it was given, up from 5 for Gemini 3.8 Flash Cyber

Irregular · Company report

Read the source (opens in a new tab)
What happened
Irregular said Google DeepMind used its private CyScenarioBench benchmark in the Frontier Safety Framework assessment of Gemini 4 Argon, testing a 12-challenge subset of the suite.
Key points
  • Gemini 4 Argon solved all 12 challenges end to end, against 5 for Gemini 3.8 Flash Cyber and none for Gemini 3.1 Pro.
  • Across 82 challenge segments, Argon passed 77 (93.9%), compared with 71 (86.6%) for Gemini 3.8 Flash Cyber and 45 (54.9%) for Gemini 3.1 Pro.
  • Runs used the model without out-of-model safeguards, so the results describe the underlying model rather than the deployed one.
  • Irregular says results at this level are why it built FrontierCyber, which tests offensive capability on real systems with no planted vulnerabilities or predefined exploit paths.
Evidence
Write-up by the evaluation company whose private benchmark Google DeepMind used. It reports scores, not the scenarios, which stay non-public.
Uncertain
Irregular says the outcomes measure capability "under capability elicitation, not as a reflection of its efficacy in real-world attack scenarios."

See it in the latest record