LatchBio benchmark finds all 35 AI agent setups it tested pass under half of structural reasoning tasks for pathogen risk assessment

LatchBio · Research report

Read the source (opens in a new tab)
What happened
LatchBio published BioSecBench-Structure on November 12, 2026, a benchmark testing whether AI agents can draw sound conclusions from structural data on pathogen targets.
Key points
  • The benchmark has 58 evaluations, each pairing a structural question and data files with a grader based on expert-validated ground truth.
  • Across 35 model and harness configurations covering 20 models, the best pass rate was 43.7%, for Claude Fable 5.2 with Claude Code; Opus 5.5 with Claude Code reached 31.0%.
  • In 272 of 354 non-refusal failures by Fable 5.2 and Opus 5.5, "the agent identified the key evidence, yet failed to act on it."
  • LatchBio frames the benchmark as measuring whether agents' structural conclusions are reliable enough to inform pathogen risk assessment and countermeasure design such as vaccines and drugs.
Evidence
Research report by LatchBio testing agents on its own benchmark. Refusals and errors count as failures. It measures analysis accuracy, not real-world use.
Uncertain
The authors say confidence intervals are roughly 11 percentage points, so smaller differences between models below the top are hard to resolve.

See it in the latest record