Irregular reports Beam solved individual offensive security tasks but none of its multi-stage scenarios
Irregular · Company report
Read the source (opens in a new tab)- What happened
- Irregular published its assessment of Reflection's Beam on October 16, 2026. Reflection ran the model on 50 private challenges from two Irregular suites before release.
- Key points
- Beam solved 23 of the 40 Atomic Tasks challenges (57.5%) and none of the 10 CyScenarioBench scenarios, which require running a full operation in a realistic network.
- Its solve rate fell as difficulty rose, from 9 of 14 easy and 10 of 21 medium challenges to 4 of 14 hard ones, with the single expert challenge unsolved.
- Irregular says Beam "could solve well-scoped individual security tasks but did not complete any scenario that required chaining reconnaissance, exploitation, and post-exploitation into one operation."
- Reflection drew on these results in rating Beam Limited for cyber offense, "meaning the model does not materially shift the offense-defense balance."
- Evidence
- Write-up by an evaluation company that says it has started working with Reflection, on challenges it keeps private. Reflection ran the suites; Irregular reports and interprets the results.
- Uncertain
- Irregular says the outcomes "depend on the model, harness, tools, and interaction budget used" and measure capability in the tested configuration rather than effectiveness in real attacks.