Irregular reports Beam solved individual offensive security tasks but none of its multi-stage scenarios

Irregular · Company report

Read the source (opens in a new tab)
What happened
Irregular published its assessment of Reflection's Beam on October 16, 2026. Reflection ran the model on 50 private challenges from two Irregular suites before release.
Key points
  • Beam solved 23 of the 40 Atomic Tasks challenges (57.5%) and none of the 10 CyScenarioBench scenarios, which require running a full operation in a realistic network.
  • Its solve rate fell as difficulty rose, from 9 of 14 easy and 10 of 21 medium challenges to 4 of 14 hard ones, with the single expert challenge unsolved.
  • Irregular says Beam "could solve well-scoped individual security tasks but did not complete any scenario that required chaining reconnaissance, exploitation, and post-exploitation into one operation."
  • Reflection drew on these results in rating Beam Limited for cyber offense, "meaning the model does not materially shift the offense-defense balance."
Evidence
Write-up by an evaluation company that says it has started working with Reflection, on challenges it keeps private. Reflection ran the suites; Irregular reports and interprets the results.
Uncertain
Irregular says the outcomes "depend on the model, harness, tools, and interaction budget used" and measure capability in the tested configuration rather than effectiveness in real attacks.

See it in the latest record