Irregular reports Gemini 4 Argon solved all 12 multi-stage offensive cyber scenarios it was given, up from 5 for Gemini 3.8 Flash Cyber
Irregular · Company report
Read the source (opens in a new tab)- What happened
- Irregular said Google DeepMind used its private CyScenarioBench benchmark in the Frontier Safety Framework assessment of Gemini 4 Argon, testing a 12-challenge subset of the suite.
- Key points
- Gemini 4 Argon solved all 12 challenges end to end, against 5 for Gemini 3.8 Flash Cyber and none for Gemini 3.1 Pro.
- Across 82 challenge segments, Argon passed 77 (93.9%), compared with 71 (86.6%) for Gemini 3.8 Flash Cyber and 45 (54.9%) for Gemini 3.1 Pro.
- Runs used the model without out-of-model safeguards, so the results describe the underlying model rather than the deployed one.
- Irregular says results at this level are why it built FrontierCyber, which tests offensive capability on real systems with no planted vulnerabilities or predefined exploit paths.
- Evidence
- Write-up by the evaluation company whose private benchmark Google DeepMind used. It reports scores, not the scenarios, which stay non-public.
- Uncertain
- Irregular says the outcomes measure capability "under capability elicitation, not as a reflection of its efficacy in real-world attack scenarios."