LatchBio benchmark finds all 35 AI agent setups it tested pass under half of structural reasoning tasks for pathogen risk assessment
LatchBio · Research report
Read the source (opens in a new tab)- What happened
- LatchBio published BioSecBench-Structure on November 12, 2026, a benchmark testing whether AI agents can draw sound conclusions from structural data on pathogen targets.
- Key points
- The benchmark has 58 evaluations, each pairing a structural question and data files with a grader based on expert-validated ground truth.
- Across 35 model and harness configurations covering 20 models, the best pass rate was 43.7%, for Claude Fable 5.2 with Claude Code; Opus 5.5 with Claude Code reached 31.0%.
- In 272 of 354 non-refusal failures by Fable 5.2 and Opus 5.5, "the agent identified the key evidence, yet failed to act on it."
- LatchBio frames the benchmark as measuring whether agents' structural conclusions are reliable enough to inform pathogen risk assessment and countermeasure design such as vaccines and drugs.
- Evidence
- Research report by LatchBio testing agents on its own benchmark. Refusals and errors count as failures. It measures analysis accuracy, not real-world use.
- Uncertain
- The authors say confidence intervals are roughly 11 percentage points, so smaller differences between models below the top are hard to resolve.