SecureBio and RAND reanalysis finds LLMs often gave novices wrong advice in a 2025 lab uplift trial

SecureBio · Research report

Read the source (opens in a new tab)
What happened
SecureBio and RAND published a reanalysis on November 6, 2026 of participant data from Active Site's 2025 randomized trial testing whether LLMs help novices complete hands-on laboratory tasks.
Key points
  • The original 8-week trial of 153 novices found overall success did not differ significantly between a group given LLMs and a group with the internet alone.
  • Reviewing transcripts, the authors found that of 55 LLM-arm participants who got advice on a key step, 18 received correct advice and 37 received incorrect advice.
  • Among the 29 who took the right approach, 12 succeeded, so success at the hands-on step "could be as high as 41%", suggesting the low uplift may partly reflect mid-2025 model errors rather than a hands-on skill barrier alone.
  • They conclude "the tacit knowledge barrier was not the only reason we didn't see novice uplift" and that the trial "should not be treated as definitive negative evidence" about current models.
Evidence
Reanalysis by a biosecurity evaluator and RAND of another group's trial data, using participant transcripts and lab notebooks. It infers an upper bound, not a measured uplift from current models.
Uncertain
The work is undergoing RAND's quality-assurance and peer review and "is subject to further revision." The authors say they do not yet have clear evidence that AI models uplift novices in the lab.

See it in the latest record